Ryan Julian
Ryan Julian is a Student Researcher for the Robotics team at Google Research and a PhD student at the Robotics and Embedded Systems Laboratory, part of the Department of Computer Science at the University of Southern California. He does research at the intersection of robotics and machine learning. His PhD advisor is Gaurav Sukhatme, and from 2017 to 2018, he was also co-advised by Stefan Schaal. His advisers at Google Research are Karol Hausman and Chelsea Finn.
From 2014 to 2017, he worked at Google and X, on a series of robotics projects, including the Everyday Robot project, whose goal is to make a robot which can "learn to help everyone, every day." He worked on many parts of the robotics stack, including high-level programming APIs, 3D visualization, interprocess communication, WiFi and cloud connectivity, automatic calibration, and automation for manufacturing lines building robots. Some of the robots he helped create can be seen in the company's earliest robot learning work.
Prior to joining Google, he was a Hardware and Computer Vision Engineer at Leap Motion (now Ultraleap), where he worked on hardware, firmware, and test automation for computer vision-based hand-tracking devices, and earned a couple patents in the process. Before that, he spent a year as a Research Scientist at UC Berkeley, where he did research with—and built controllers for—some of the world's smallest intelligent robots.
Ryan has a BS in EECS from UC Berkeley, where he worked with Ron Fearing at the Biomimetic Millisystems Lab.
Research Areas
Authored Publications
Sort By
Deep RL at Scale: Sorting Waste in Office Buildings with a Fleet of Mobile Manipulators
Jarek Rettinghouse
Daniel Ho
Julian Ibarz
Sangeetha Ramesh
Matt Bennice
Alexander Herzog
Chuyuan Kelly Fu
Adrian Li
Kim Kleiven
Jeff Bingham
Yevgen Chebotar
David Rendleman
Wenlong Lu
Mohi Khansari
Mrinal Kalakrishnan
Ying Xu
Noah Brown
Khem Holden
Justin Vincent
Peter Pastor Sampedro
Jessica Lin
David Dovo
Daniel Kappler
Mengyuan Yan
Sergey Levine
Jessica Lam
Jonathan Weisz
Paul Wohlhart
Karol Hausman
Cameron Lee
Bob Wei
Yao Lu
Preview abstract
We describe a system for deep reinforcement learning of robotic manipulation skills applied to a large-scale real-world task: sorting recyclables and trash in office buildings. Real-world deployment of deep RL policies requires not only effective training algorithms, but the ability to bootstrap real-world training and enable broad generalization. To this end, our system combines scalable deep RL from real-world data with bootstrapping from training in simulation, and incorporates auxiliary inputs from existing computer vision systems as a way to boost generalization to novel objects, while retaining the benefits of end-to-end training. We analyze the tradeoffs of different design decisions in our system, and present a large-scale empirical validation that includes training on real-world data gathered over the course of 24 months of experimentation, across a fleet of 23 robots in three office buildings, with a total training set of 9527 hours of robotic experience. Our final validation also consists of 4800 evaluation trials across 240 waste station configurations, in order to evaluate in detail the impact of the design decisions in our system, the scaling effects of including more real-world data, and the performance of the method on novel objects.
View details
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
Alexander Herzog
Alexander Toshkov Toshev
Andy Zeng
Anthony Brohan
Brian Andrew Ichter
Byron David
Chelsea Finn
Clayton Tan
Diego Reyes
Dmitry Kalashnikov
Eric Victor Jang
Jarek Liam Rettinghouse
Jornell Lacanlale Quiambao
Julian Ibarz
Karol Hausman
Kyle Alan Jeffrey
Linda Luu
Mengyuan Yan
Michael Soogil Ahn
Nicolas Sievers
Noah Brown
Omar Eduardo Escareno Cortes
Peng Xu
Peter Pastor Sampedro
Rosario Jauregui Ruano
Sally Augusta Jesmonth
Sergey Levine
Steve Xu
Yao Lu
Yevgen Chebotar
Yuheng Kuang
Conference on Robot Learning (CoRL) (2022)
Preview abstract
Large language models can encode a wealth of semantic knowledge about the world. Such knowledge could in principle be extremely useful to robots aiming to act upon high-level, temporally extended instructions expressed in natural language.
However, a significant weakness of language models is that they lack contextual grounding, which makes it difficult to leverage them for decision making within a given real-world context.
For example, asking a language model to describe how to clean a spill might result in a reasonable narrative, but it may not be applicable to a particular agent, such as a robot, that needs to perform this task in a particular environment.
We propose to provide this grounding by means of pretrained behaviors, which are used to condition the model to propose natural language actions that are both feasible and contextually appropriate.
The robot can act as the language model’s “hands and eyes,” while the language model supplies high-level semantic knowledge about the task.
We show how low-level tasks can be combined with large language models so that the language model provides high-level knowledge about the procedures for performing complex and temporally extended instructions, while value functions associated with these tasks provide the grounding necessary to connect this knowledge to a particular physical environment.
We evaluate our method on a number of real-world robotic tasks, where we show that this approach is capable of executing long-horizon, abstract, natural-language tasks on a mobile manipulator.
The project's website and the video can be found at \url{say-can.github.io}.
View details
Actionable Models: Unsupervised Offline Learning of Robotic Skills
Benjamin Eysenbach
Chelsea Finn
Dmitry Kalashnikov
Jake Varley
Karol Hausman
Sergey Levine
Yao Lu
Yevgen Chebotar
International Conference on Machine Learning 2021 (2021)
Preview abstract
We consider the problem of learning useful robotic skills from previously collected offline data without access to manually specified rewards or additional online exploration, a setting that is becoming increasingly important for scaling robot learning by reusing past robotic data. In particular, we propose the objective of learning a functional understanding of the environment by learning to reach any goal state in a given dataset. We employ goal-conditioned Q-learning with hindsight relabeling and develop several techniques that enable training in a particularly challenging offline setting. We find that our method can operate on high-dimensional camera images and learn a variety of skills on real robots that generalize to previously unseen scenes and objects. We also show that our method can learn to reach long-horizon goals across multiple episodes, and learn rich representations that can help with downstream tasks through pre-training or auxiliary objectives.
View details
Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement Learning
Benjamin Swanson
Gaurav Sukhatme
Sergey Levine
Chelsea Finn
Karol Hausman
Conference on Robot Learning, PMLR (2020)
Preview abstract
One of the great promises of robot learning systems is that they will be able to learn from their mistakes and continuously adapt to ever-changing environments. Despite this potential, most of the robot learning systems today are deployed as a fixed policy and they are not being adapted after their deployment. Can we efficiently adapt previously learned behaviors to new environments, objects and percepts in the real world? In this paper, we present a method and empirical evidence towards a robot learning framework that facilitates continuous adaption. In particular, we demonstrate how to adapt vision-based robotic manipulation policies to new variations by fine-tuning via off-policy reinforcement learning, including changes in background, object shape and appearance, lighting conditions, and robot morphology. Further, this adaptation uses less than 0.2% of the data necessary to learn the task from scratch. We find that our approach of adapting pre-trained policies leads to substantial performance gains over the course of fine-tuning, and that pre-training via RL is essential: training from scratch or adapting from supervised ImageNet features are both unsuccessful with such small amounts of data. We also find that these positive results hold in a limited continual learning setting, in which we repeatedly fine-tune a single lineage of policies using data from a succession of new tasks. Our empirical conclusions are consistently supported by experiments on simulated manipulation tasks, and by 52 unique fine-tuning experiments on a real robotic grasping system pre-trained on 580,000 grasps.
View details
Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
Tianhe Yu
Zhanpeng He
Deirdre Quillen
Karol Hausman
Chelsea Finn
Sergey Levine
Conference on Robot Learning (2019)
Preview abstract
Meta-reinforcement learning algorithms can enable robots to acquire
new skills much more quickly, by leveraging prior experience to learn how to
learn. However, much of the current research on meta-reinforcement learning focuses on task distributions that are very narrow. For example, a commonly used
meta-reinforcement learning benchmark uses different running velocities for a
simulated robot as different tasks. When policies are meta-trained on such narrow
task distributions, they cannot possibly generalize to more quickly acquire entirely
new tasks. Therefore, if the aim of these methods is enable faster acquisition of
entirely new behaviors, we must evaluate them on task distributions that are sufficiently broad to enable generalization to new behaviors. In this paper, we propose
an open-source simulated benchmark for meta-reinforcement learning and multitask learning consisting of 50 distinct robotic manipulation tasks. Our aim is to
make it possible to develop algorithms that generalize to accelerate the acquisition
of entirely new, held-out tasks. We evaluate 6 state-of-the-art meta-reinforcement
learning and multi-task learning algorithms on these tasks. Surprisingly, while
each task and its variations (e.g., with different object positions) can be learned
with reasonable success, these algorithms struggle to learn with multiple tasks at
the same time, even with as few as ten distinct training tasks. Our analysis and
open-source environments pave the way for future research in multi-task learning
and meta-learning that can enable meaningful generalization, thereby unlocking
the full potential of these methods
View details
Scaling simulation-to-real transfer by learning composable robot skills
Eric Heiden
Zhanpeng He
Hejia Zhang
Stefan Schaal
Joseph Lim
Gaurav Sukhatme
Karol Hausman
International Symposium on Experimental Robotics (ISER) 2018 (2018)
Preview abstract
We present a novel solution to the problem of simulation-to-real transfer, which builds on recent advances in robot skill decomposition. Rather than focusing on minimizing the simulation-reality gap, we learn a set of diverse policies that are parameterized in a way that makes them easily reusable. This diversity and parameterization of low-level skills allows us to find a transferable policy that is able to use combinations and variations of different skills to solve more complex, high-level
tasks.
In particular, we first use simulation to jointly learn a policy for a set of low-level skills, and a “skill embedding” parameterization which can be used to compose them. Later, we learn high-level policies which actuate the low-level policies via this skill embedding parameterization. The high-level policies encode how and when to reuse the low-level skills together to achieve specific high-level tasks. Importantly, our method learns to control a real robot in joint-space to achieve these high-level
tasks with little or no on-robot time, despite the fact that the low-level policies may not be perfectly transferable from simulation to real, and that the low-level skills were not trained on any examples of high-level tasks.
We illustrate the principles of our method using informative simulation experiments. We then verify its usefulness for real robotics problems by learning, transferring, and composing free-space and contact motion skills on a Sawyer robot using only joint-space control. We experiment with several techniques for composing pre-learned skills, and find that our method allows us to use both learning-based approaches and efficient search-based planning to achieve high-level tasks using only pre-learned skills.
View details
