Mind the GAP: Geometry Aware Passthrough mitigates cybersickness

Virtual reality (VR) headsets are becoming increasingly popular, especially in environments that benefit from immersion, such as gaming and education. In part, they accomplish this by restricting the user's perception to the virtual world that they are projecting. Yet many headsets, such as Meta Quest, Apple Vision Pro, and Samsung Moohan, also give the users the option to view their real-world surroundings. These video see-through (VST) headsets accomplish this with a passthrough system that uses world-facing cameras.

As the adoption of VST devices increases, so does the need to address the discomfort and cybersickness experienced by some users of VST technology. While motion sickness in VR has been extensively studied in the last few decades, there is limited work dedicated to enhancing comfort and safety with the use of VST headsets or other augmented reality (AR) devices. Insights from VR research can be informative, but the unique experience of VST, wherein users can see and interact with the physical world, warrants dedicated investigation to guide the design of VST head-mounted displays (HMD) that are comfortable for users.

In “Mind the GAP: Geometry Aware Passthrough Mitigates Cybersickness” (to be presented at CHI 2025), we demonstrate the potential of GAP systems to mitigate cybersickness through accurate depth perception. We propose a protocol to quantitatively measure cybersickness experienced by users in VST headsets. Using this protocol, we conduct a user study to compare DP and GAP systems. To the best of our knowledge, our study is the first one to reveal significantly reduced nausea, disorientation, and overall scores of cybersickness with GAP. It also uncovers several potential avenues to further mitigate visually-induced discomfort.

Geometry Aware Passthrough

Due to inherent hardware limitations common in VST headsets, the camera's perspective deviates from the user's natural viewpoint. Direct passthrough (DP) delivers the raw camera feed to the display, which exaggerates distances and consequently the motion of objects. Compared to natural vision (NV), DP results in visual artifacts such as disocclusion, inaccurate perception of object positions, and exaggerated motion parallax. According to the sensory-conflict theory, such a mismatch between visual and inertial cues can potentially cause discomfort.

To mitigate this mismatch between DP and natural vision, past research has focused on utilizing geometry information to reproject camera feeds into the natural view from the eyes. We introduce the term geometry aware passthrough (GAP) to describe these passthrough systems. While previous work assumes that a GAP reduces discomfort compared to DP, to the best of our knowledge, no empirical studies have directly investigated or verified this assumption.

Geometrical accuracy and distortion detection

Our analysis also involves a rigorous quantitative assessment of the geometrical accuracy and warping introduced by each system. This includes evaluating how accurately each system represents real-world geometry (position and scale of objects) within the AR environment and whether distortions get introduced in the passthrough video feed (as shown in the figure above). More details are in the paper.

User study design

We introduce a comprehensive protocol focused on key VST use cases to holistically assess visually-induced discomfort and cybersickness in VST HMDs. We then use this to compare our GAP algorithm to DP.

To achieve reproducibility, repeatability, and real-life relevance, we began with tasks identified in the literature, tested them in a pilot study, and iteratively refined the task nature and duration based on participant feedback. A total of 25 consenting participants with normal or corrected-to-normal vision completed the tasks for the study. Each participant experienced all conditions including NV, DP, and GAP, allowing for a direct comparison of each participant’s experience.

We devised our protocol focusing exclusively on passthrough-based real-world interactions and ensured no virtual elements were visible to participants. The tasks were inspired from fundamental real-world XR applications such as working with laptops for productivity, navigation in the physical world, and interaction with real-world objects. They emphasized user head motion while necessitating inspection and spatial awareness of the physical world. Specifically:

• Typing: This task was chosen to reflect emerging applications in productivity and to effectively engage both visual and motor components. Participants typed on a physical Dvorak keyboard, chosen because it required frequent gaze shifts between the keyboard and a laptop screen.
• Navigation: The realistic and holistic use of a VST HMD involves navigating physical spaces, avoiding obstacles, and interacting with real-world objects. We designed a navigation task where participants collected and dropped off 10 numbered cones, one at a time, into a designated drop zone. This task emphasized geometry perception and required multi-directional movements.
• Interaction: This task was designed to simulate common assembly tasks requiring both motor and cognitive skills. Participants assembled large 24-piece jigsaw puzzles by retrieving and working on only one batch of 8 puzzle pieces at a time within a rectangular frame marked on the table. The large puzzle size was selected to accommodate head motion which is often associated with motion sickness.

To measure cybersickness, we used the Simulator Sickness Questionnaire (SSQ), which categorizes symptoms along four subscales: nausea, disorientation, oculomotor, and total severity. Participants filled out the SSQ before and after each task to isolate the cybersickness experienced in each mode. In addition to the SSQ, participants rated their general discomfort on a scale of 0 to 10 after completing each task. Finally, participants provided qualitative feedback on their experiences at the end of each task.

Findings: GAP significantly reduces cybersickness

We found that GAP significantly reduced cybersickness compared to DP. Specifically, GAP led to lower scores in the nausea, disorientation, and total severity subscales of the SSQ (𝑝 < 0.05); lower discomfort scores across all tasks compared to DP, including typing (𝑝 = 0.046), navigation (𝑝 = 0.041), and interaction (𝑝 = 0.022); as well as significantly lower average discomfort scores (𝑝 = 0.016). In line with previous work on VST, our results revealed a VST symptom profile for cybersickness that is distinguished from other types of motion sickness. The most reported symptoms for both DP and GAP were sweating, eyestrain, general discomfort, and headache.

Participants also generally preferred GAP over DP subjectively. Several participants reported a mismatch between vision and motion with DP, aligning with the sensory-conflict theory. This was frequently mentioned in the context of head motion, particularly during the interaction task. DP caused impaired spatial awareness compared to GAP, and participants noted that DP caused them to move closer to objects due to inaccurate depth cues. Some users also experienced unstable gait and collisions with furniture while using DP. While most participants preferred GAP, some expressed a preference for DP for certain tasks such as typing due to the warping artifacts of GAP on the keyboard. However, some participants adapted to these artifacts over time.

Implications for future VST design

Our findings imply that VST HMD design should incorporate GAP to improve user comfort and facilitate the adoption of VST technology. However, GAP introduces additional computational demands compared to DP. Future design efforts should strive for a balance that maintains the benefits of GAP while minimizing the impact on system performance and latency.

Participant feedback also helped identify several areas for future research into enhancing comfort in VST. Four issues emerged: frame drops, overexposed images, latency, and blurry vision. Several participants reported that slight delays when moving their heads caused nausea and discomfort. This suggests the need for more research to understand the impact of these factors on mitigating user discomfort and cybersickness.

Conclusion

Mind the GAP is the first work that demonstrates that GAP significantly reduces nausea, disorientation, and total scores of cybersickness as well as subjective discomfort scores as compared to DP. We present a comprehensive protocol aimed at evaluating visually-induced discomfort and cybersickness in VST HMDs through key use cases. We hope that our comprehensive protocol sets a foundation for future studies aimed at refining these systems and enhancing user comfort in VST technologies.

Acknowledgments

This research was conducted by Trishia El Chemaly during her time at Google as a Student Researcher, Mohit Goyal, Tinglin Duan, Vrushank Phadnis, Sakar Khattar, Bjorn Vlaskamp, Achin Kulshrestha, Eric Lee Turner, Aveek Purohit, Gregory Neiswander, and Konstantine Tsotsos. We would also like to thank Abhishek Kar for his guidance and help during the ideation of this work.