About

I am a CS PhD student at Stanford, advised by Chelsea Finn, and a research intern at Physical Intelligence (Pi). Previously, I worked with Yuke Zhu and Jim Fan at NVIDIA Research. I received my Bachelor’s degree in CS from USC, advised by Joseph J. Lim and Youngwoon Lee.

I am broadly interested in robot learning. My research goal is to create general-purpose robots that seamlessly perform complex, long-horizon tasks in our daily lives — from homes to factories, handling both the tedious and the dangerous. I am drawn to developing simple methods that scale well with compute and data.

I deeply believe in human ingenuity and the potential of AI. My goal for the next 20 years is to become a university professor and I want to establish the next Bell Labs — the idea factory that revolutionized the world for the better.

News

  • [Nov, 2024] Co-organizing the Workshop on Cross-Embodiment Robot Learning at CoRL 2024
  • [Oct, 2024] Our work on general-purpose robot foundation model was featured in WIRED, The New York Times, etc. See the paper and blog post for more details
  • [Sep, 2024] I started my PhD, supported by Stanford SoE Fellowship
  • [May, 2024] One paper is accepted to RSS 2024
  • [Mar, 2024] I joined Physical Intelligence (π) as the first intern :)
  • [Mar, 2024] Media coverage on YAY Robot
  • [Sep, 2023] One paper is accepted to NeurIPS 2023
  • [Sep, 2023] One paper is accepted to CoRL 2023
  • [Aug, 2023] Invited talk at Google DeepMind on AWE
  • [Feb, 2023] Invited talk at MILA on skill-based and model-based RL
  • [Sep, 2022] One paper is accepted to CoRL 2022

Research

publication teaser

Yell At Your Robot: Improving On-the-Fly from Language Corrections


Lucy Xiaoyang Shi, Zheyuan Hu, Tony Z. Zhao, Archit Sharma, Karl Pertsch, Jianlan Luo, Sergey Levine, Chelsea Finn
Robotics: Science and Systems (RSS), 2024
paper / project / code / twitter /

YAY Robot leverages verbal corrections to enable on-the-fly adaptation and continuous policy improvement on complex long-horizon tasks.

publication teaser

Cross-Episodic Curriculum for Transformer Agents


Lucy Xiaoyang Shi*, Yunfan Jiang*, Jake Grigsby, Linxi 'Jim' Fan†, Yuke Zhu†
Neural Information Processing Systems (NeurIPS), 2023
paper / project / code / twitter /

CEC enhances Transformer agents’ learning efficiency and generalization by structuring cross-episodic experiences in-context.

publication teaser

Waypoint-Based Imitation Learning for Robotic Manipulation


Lucy Xiaoyang Shi*, Archit Sharma*, Tony Z. Zhao, Chelsea Finn
Conference on Robot Learning (CoRL), 2023
paper / project / code / twitter /

We propose an automatic method for extracting waypoints from demonstrations for performant imitation learning.

publication teaser

Skill-based Model-based Reinforcement Learning


Lucy Xiaoyang Shi, Joseph J. Lim, Youngwoon Lee
Conference on Robot Learning (CoRL), 2022
paper / project / code / twitter /

We devise a method that enables model-based RL on long-horizon, sparse-reward tasks, allowing us to learn with 5x less samples.


Other Projects (Technical Reports/Talks)

  • Human Data Augmentation for Human-Robot Collaboration, advised by Prof. Stefanos Nikolaidis, 2022
  • Unsupervised Exploration with Episodic Memory, advised by Siyuan Li and Prof. Chongjie Zhang, 2021
  • Quantum Mechanics for Next-Gen Spacecrafts, advised by Dr. Rahul Rughani and Prof. David Barnhart, 2020
  • Avionics R&D in Rocketry, advised by Dr. Paul Giuliano, 2019-2021
  • (Subscale) Nuclear Thermal Rocket Engine, advised by Dr. Charles Radovich and Dr. Matthew Gilpin, 2019-2020

A mountain of gratitude for people who have kindly mentored me and inspired me with their vision and passion!


Research Interests

Within deep reinforcement learning, embodied AI, and robotics, I am particularly interested in:

Model-based learning: build internal world models of the embodied self and the world to develop a general understanding and improve sample-efficiency in learning through planning.

Hierarchical / Skill-based learning: enable long-horizon task learning through temporal abstraction and guided exploration.

Representation learning: autonomously infer and discover the underlying temporal, spatial, and causal structures of the input across different modalities for efficient decision-making and generalization.

A sample of other topics that I’m also very curious about: generative models, external memory, and learning from human videos.

Teaching and Mentoring

Misc

In my free time, I enjoy reading, bouldering, and hiking. I also like exploring new places, primarily through casual runs or getting lost in the wild. I love living everyday as an adventure — and this is especially the case when I invite friends to try the novel-flavored cookies I just baked. :)

Below I expand a little bit on some fun experiences that have shaped the way I think.

Envision a home on Mars

mars teaser

With my two brilliant friends Jessica Yuan and Albert Sun we published a paper exploring a Mars city state of 1 million people. We were the only undergraduate team to win the finalist (10 /176) and present at the 23rd Annual International Mars Society Convention. The paper was also later included as a chapter in the book, Mars City States: New Societies for a New World, published in 2021.

Looking back, it was a wonderful summer of reading and thinking intensively to paint a coherent picture of the future. It gave me an epiphany that a myriad of things in the world are profoundly connected, and AI will be an important driving force behind many things I care about — like an interplanetary humanity, robotics, and universal education.

Schrödinger’s me

I once worked with a group of physicist and engineers to experimentally analyze the quantum theory quantized inertia as an alternative to dark matter. As far as I know the experiments haven’t been fruitful so far and we still have no proof / disproof for this quantum theory or even the dark matter. But I had a ton of fun learning about puzzles in quantum physics along the way! This experience also trained me to question the fundamental and think from first principles.

Literally “Rocket Science”


rpl teaser

In parallel to the physics stuff I was also an avionics engineer at the USC Rocket Propulsion Lab (think: build brains for rockets). We are the first student group to successfully launch and recover an entirely student-designed rocket (Traveler IV) past the Karman line, the recognized boundary of space at 100 km (328,084 ft)! There were so many difficulties to overcome, but launching a rocket in desert is one of my best college memories! I love this group of people — they are ambitious, hungry for getting things to work, and incredibly caring. They also taught me if you care about something deeply enough, the reality becomes negotiable.


A Quest for Meaning

I wasted a lot of misguided, youthful time thinking about the following three questions:

  1. Why does the universe exist?
  2. Why does humanity exist?
  3. Why do I exist?

I like reading philosophy, which humbles and inspires me (fun fact: I once majored in it!). So don’t be too surprised if we suddenly start discussing the meaning of life in a casual conversation (I won’t make it scary, I promise). ;)

However, now instead of analyzing the limit of reasoning, I want to use my ability of reasoning to push the limit of reasoning. At present Artificial Intelligence is my best answer.