Johnny Chung Lee
I joined Google in 2011, and I am currently an Engineering Director in the AR/XR team focused on multi-modal AI systems for wearable devices, such as Project Astra. My areas of interest include machine learning, computer vision, sensors, robotics, and human-computer interaction. In my career, I've had the opportunity to work on the following projects:
- Founded a program in the Google Research/Deepmind Robotics program focusing on teleoperation and learning from human demonstrations. The team developed a production operational micro-fulfillment warehouse using robotic arms for pick & pack from an automatic storage and retrieval system (ASRS).
- Founded and lead Google Tango in 2013, a mobile computer vision effort to bring 6-DOF tracking and 3D sensing to mobile phones, tablets, and headsets. This included the development and launch of multiple mobile phones incorporating 3D sensing and tracking. The technology from Tango has become ARCore for Android enabling mobile AR applications running on billions of phones, and the Visual Positioning Service for Google Maps Geospatial API enabling global scale precision location and immersive content.
- Lead the software team that developed 6-DOF tracking for Daydream VR headsets and future XR hardware.
- Oversaw some hardware R&D efforts for VR/AR technologies, including novel tracking technologies, headset prototypes, optical technologies, and sensing algorithms.
- As a part of Google X, I made early contributions to Google Glass, Loon, and other non-public projects.
- Prior to Google, I developed human tracking algorithms for the Xbox 360 Kinect at Microsoft.
- Personal projects have included low-cost interactive whiteboards and desktop VR displays using the Nintendo Wii remote, which have received over 15 million views, performed as a TED talk in 2008, and recognized by MIT Technology Review's TR35.
Authored Publications
Sort By
Google
InstructPipe: Generating Visual Blocks Pipelines with Human Instructions and LLMs
Zhongyi Zhou
Jing Jin
Xiuxiu Yuan
Jun Jiang
Jingtao Zhou
Yiyi Huang
Kristen Wright
Jason Mayes
Mark Sherwood
Alex Olwal
Ram Iyengar
Na Li
Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI), ACM, pp. 23
Augmented Object Intelligence with XR-Objects
Mustafa Doga Dogan
Karan Ahuja
Andrea Colaco
Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (UIST), ACM (2024), pp. 1-15
Experiencing InstructPipe: Building Multi-modal AI Pipelines via Prompting LLMs and Visual Programming
Zhongyi Zhou
Jing Jin
Xiuxiu Yuan
Jun Jiang
Jingtao Zhou
Yiyi Huang
Kristen Wright
Jason Mayes
Mark Sherwood
Alex Olwal
Ram Iyengar
Na Li
Extended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems, ACM, pp. 5
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language
Andy Zeng
Brian Ichter
Stefan Welker
Aveek Purohit
Michael Ryoo
Pete Florence
arXiv (2022)
Learning to Fold Real Garments with One Arm: A Case Study in Cloud-Based Robotics Research
Ryan Hoque
Kaushik Shivakumar
Shrey Aeron
Gabriel Deza
Aditya Ganapathi
Andy Zeng
Ken Goldberg
IEEE International Conference on Intelligent Robots and Systems (IROS) (2022) (to appear)
Implicit Behavioral Cloning
Pete Florence
Corey Lynch
Andy Zeng
Oscar Ramirez
Laura Downs
Igor Mordatch
CoRL (2021)
Transporter Networks: Rearranging the Visual World for Robotic Manipulation
Andy Zeng
Pete Florence
Stefan Welker
Jonathan Chien
Travis Armstrong
Ivan Krasin
Dan Duong
Conference on Robot Learning (CoRL) (2020)
ClearGrasp: 3D Shape Estimation of Transparent Objects for Manipulation
Shreeyak Sajjan
Matthew Moore
Mike Pan
Ganesh Nagaraja
Andy Zeng
Shuran Song
IEEE International Conference on Robotics and Automation (ICRA) (2020)
Grasping in the Wild: Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrations
Shuran Song
Andy Zeng
IEEE Robotics and Automation Letters (RA-L) (2020)
Spatial Action Maps for Mobile Manipulation
Jimmy Wu
Xingyuan Sun
Andy Zeng
Shuran Song
Szymon Rusinkiewicz
Robotics: Science and Systems (RSS) (2020)
