Mar Gonzalez-Franco
Mar Gonzalez-Franco, PhD, is a Computer Scientist and Neuroscientist at Google working on a new generation of Immersive technologies. With a background in real-time systems in her research she tries to build better interactions for immersive technologies using different disciplines: Virtual Reality, Augmented Reality, AI, computer graphics, computer vision, Avatars, and haptics. All while studying human behavior, perception and neuroscience.
She was awarded the 2022 IEEE VGTC VR New Researcher Award, and the NAE Frontiers Engineer.
She leads the BIRD lab, working on Blended Intelligence Research and Devices.
Research Areas
Authored Publications
Sort By
Preview abstract
Mid-air gestures in Extended Reality (XR) often lead to fatigue, discomfort and imprecision, limiting their suitability for extended use. Surface-based interactions offer a compelling alternative, providing improved accuracy, speed, and comfort. However, current egocentric vision-based methods struggle with reliable surface inputs due to challenges in hand tracking and surface-plane estimation from oblique and occluded viewing angles. To this extent, we introduce SurfaceXR, a novel sensor fusion approach that combines headset based hand tracking with micro-vibration data sampled from commodity smartwatch IMUs to enable precise and robust inputs on arbitrary surfaces. Our system is designed with flexibility in mind - it can function using only hand tracking, only IMU sensing, or optimally with both modalities combined. Our user study across 12 participants validates SurfaceXR's effectiveness in augmenting surface touch tracking and 8 class hand-surface gesture recognition, demonstrating significant improvements over single-modality approaches. Enabled by SurfaceXR, we demonstrate a series of interactive apps for both AR and VR, ranging from on-surface sketching, text entry and gesture based navigation.
View details
Text Entry for XR Trove (TEXT): Collecting and Analyzing Techniques for Text Input in XR
Arpit Bhatia
Moaaz Hudhud Mughrabi
Massimiliano Di Luca
Diar Abdlkarim
Karan Ahuja
Hasti Seifi
ACM CHI (2025)
Preview abstract
Text entry for extended reality (XR) is far from perfect, and a variety of text entry techniques (TETs) have been proposed to fit various contexts of use. However, comparing between TETs remains challenging due to the lack of a consolidated collection of techniques, and limited understanding of how interaction attributes of a technique (e.g., presence of visual feedback) impact user performance.
To address these gaps, this paper examines the current landscape of XR TETs by creating a database of 176 different techniques. We analyze this database to highlight trends in the design of these techniques, the metrics used to evaluate them, and how various interaction attributes impact these metrics. We discuss implications for future techniques and present TEXT: Text Entry for XR Trove, an interactive online tool to navigate our database.
View details
ForcePinch: Force-Responsive Spatial Interaction for Tracking Speed Control and Beyond
Chenyang Zhang
Tiffany S Ma
John Andrews
Yalong Yang
ACM UIST (2025)
Preview abstract
Spatial interaction in 3D environments requires balancing efficiency and precision, which requires dynamic tracking speed adjustments. However, existing techniques often couple tracking speed adjustments directly with hand movements, reducing interaction flexibility. Inspired by the natural friction control inherent in the physical world, we introduce ForcePinch, a novel force-responsive spatial interaction method that enables users to intuitively modulate pointer tracking speed and smoothly transition between rapid and precise movements by varying their pinching force. To implement this concept, we developed a hardware prototype integrating a pressure sensor with a customizable mapping function that translates pinching force into tracking speed adjustments. We conducted a user study with 20 participants performing well established 1D, 2D, and 3D object manipulation tasks, comparing ForcePinch against distance-responsive technique Go-Go and speed-responsive technique PRISM. Results highlight distinctive characteristics of the force-responsive approach across different interaction contexts. Drawing insights from these findings, we further propose a generative framework designed to systematically extend force-responsive interactions to diverse spatial interaction scenarios. To illustrate the versatility and practical applicability of this framework, we designed and implemented eight representative applications, offering guidelines and inspiration for future force-enabled interaction designs.
View details
H2E: Hand, Head, Eye: A Multimodal Cascade of Natural Inputs
Khushman Patel
Hans Gellersen
Ken Pfeuffer
IEEE VR (2025)
Preview abstract
Eye-based interaction techniques for extended reality, such as gaze and pinch, are simple to use however suffer from input precision issues. We present H2E, a fine and coarse-grained pointing technique that cascades Hand, Head, and Eye inputs. As users initiate a pinch gesture, a cursor appears at the gaze point that can be dragged by head pointing before pinch confirmation. This has the potential advantage that it can add a precision component without changing the semantics of the technique. In this paper, we describe the design and implementation of the technique. Furthermore, we present an evaluation of our method in a Fitts-based user study, exploring the speed-accuracy trade-offs against a gaze and pinch interaction baseline.
View details
Online-EYE: Multimodal Implicit Eye Tracking Calibration for XR
Baosheng James Hou
Lucy Abramyan
Prasanthi Gurumurthy
Khushman Patel
Haley Adams
Andrea Colaco
Ken Pfeuffer
Hans Gellersen
Karan Ahuja
2025
Preview abstract
Unlike other inputs for VR that work out of the box, eye tracking typically requires custom calibration per user or session. We present a multimodal inputs approach for implicit calibration of eye tracker in VR, leveraging UI interaction for continuous, background calibration. Our method analyzes gaze data alongside controller interaction with UI elements, and employing ML techniques it continuously refines the calibration matrix without interrupting users from their current tasks. Potentially eliminating the need for explicit calibration. We demonstrate the accuracy and effectiveness of this implicit approach across various tasks and real time applications achieving comparable eye tracking accuracy to native, explicit calibration.
View details
Preview abstract
XR devices running chatbots powered by Large Language Models (LLMs) have the potential to become always-on agents that significantly enhance productivity scenarios. Current screen-based chatbots fail to fully utilize the comprehensive suite of natural inputs available in XR, including inward-facing sensor data. Instead, they over-rely on explicit voice or text prompts, sometimes paired with multi-modal data included in the query. We propose a solution that leverages an attention framework to implicitly derive context from user actions, eye gaze, and contextual memory within the XR environment. Our approach minimizes the need for explicitly engineered prompts, fostering intuitive and grounded interactions that provide deeper user insights for the chatbot.
View details
Preview abstract
We present HandOver, a new interaction technique designed to unify precise mouse‐based selection with intuitive hand‐tracking for virtual object manipulation. Our system’s depth‐aware 3D cursor allows users to join the accuracy of mouse input for near‐field tasks, then seamlessly switch to a “hand clone” when the physical hand lifts off the mouse area. This transition mechanism extends direct grasping and ray‐casting capabilities to distant objects, mitigating the fatigue and jitter often associated with purely gestural input. In a formal user study, we compare HandOver against two ray‐based techniques: Ray+Hand and Ray—across near, mid, and far distances. Quantitative results indicate that HandOver maintains minimal error and faster targeting times at far‐field ranges while offering fluid transitions into more distant interactions. Subjective feedback and ergonomic measures (e.g., RULA posture, NASA‐TLX workload) further reinforce that blending mouse precision with hands‐on expressiveness can improve user comfort, accuracy, and speed in immersive 3D environments. Bridging these paradigms yields a single, continuous workflow that adapts to different spatial contexts, reaffirming the role of “the hand” in modern virtual object manipulation.
View details
Preview abstract
Interacting with real-world objects in Mixed Reality (MR) often proves difficult when they are crowded, distant, or partially occluded, hindering straightforward selection and manipulation. We observe that these difficulties stem from performing interaction directly on physical objects, where input is tightly coupled to their physical constraints. Our key insight is to decouple interaction from these constraints by introducing proxies–abstract representations of real-world objects. We embody this concept in Reality Proxy, a system that seamlessly shifts interaction targets from physical objects to their proxies during selection. Beyond facilitating basic selection, Reality Proxy uses AI to enrich proxies with semantic attributes and hierarchical spatial relationships of their corresponding physical objects, enabling novel and previously cumbersome interactions in MR -such as skimming, attribute-based filtering, navigating nested
groups, and complex multi-object selections— all without requiring new gestures or menu systems. We demonstrate Reality Proxy’s versatility across diverse scenarios, including office information retrieval, large-scale spatial navigation, and multi-drone control.
An expert evaluation suggests the system’s utility and usability, suggesting that proxy-based abstractions offer a powerful and generalizable interaction paradigm for future MR systems.
View details
Beyond the Phone: Exploring Context-aware Interaction Between Mobile andMixed Reality Devices
Fengyuan Zhu
Daniel Kalmar
Mahdi Tayarani
2025
Preview abstract
Despite the surge in popularity of virtual reality (VR), mobile phones remain the primary medium for accessing digital content, offering both privacy and portability. This short paper presents Beyond the Phone, a novel framework that enhances mobile phones in VR with context-aware controls and spatial augmentation. We first establish a comprehensive design space through brainstorming and iterative discussions with VR experts. We then develop a proof-of-concept system that analyzes UI layouts to offer context-aware controls and spatial augmentation, targeting six key application areas within our design space. Finally, we demonstrate that our system can effectively adapt to a broad spectrum of applications at runtime, and discuss future directions with reviews with seven experts.
View details
Preview abstract
The question of whether AI possesses human-like experience is perhaps misleading, as it’s too abstract to tackle as a whole. Indeed humans exhibit many layers of complex feelings and emotions, so it is probably better to explore, one by one, different human traits. For example, recent studies have shown some AI chatbots can apparently show more empathy towards coworkers than their own peers. And yes, we know some people lack empathy; perhaps that isn’t the news. However, one common trait of humans everywhere is their capacity to suffer. That is the aspect on which we will focus on this paper: how is human suffering different or similar to AI?
Suffering has captured the attention of AI research in the past. Here we go one step further, giving it a neuroscientific revision along with the groundings in Bayesian inference, behavioral psychology, and theories of consciousness.
View details
