Researchers from The Control and Learning for Autonomous Robotics (CLeAR) Group have introduced Simulation Distillation (SimDist), a new framework for real-world robot adaptation based on world models. In many robotic systems, policies trained in simulation can fail after deployment because real-world dynamics differ from those seen in training. Directly finetuning those policies in the real world is often unreliable: it requires costly trial-and-error on limited data and can lead to performance collapse instead of improvement. SimDist addresses this by replacing direct policy finetuning with a more structured and stable adaptation process.
SimDist addresses this by pretraining a latent world model in simulation, distilling key task structure—such as representations, rewards, and future value—into components that can be transferred to the real robot. At deployment, the robot uses the world model for online planning, evaluating candidate actions based on predicted outcomes, while adapting only the dynamics model to match real-world behavior. By preserving task structure learned in simulation and restricting adaptation to short-horizon dynamics prediction, SimDist turns real-world improvement into a lightweight system-identification problem rather than relearning behavior from scratch. The result is a more stable and data-efficient way to recover from sim-to-real failures and rapidly improve performance on real robots.
Across precise manipulation and quadrupedal locomotion tasks, SimDist substantially outperforms prior approaches in data efficiency, stability, and final performance, achieving strong gains with only 15–30 minutes of real-world data. More broadly, the results suggest that embedding simulation-derived structure into a planning-oriented world model offers a practical path toward more scalable and reliable deployment of learned control policies.
Authors: Jacob Levy, Tyler Westenbroek, Kevin Huang, Fernando Palafox, Patrick Yin, Shayegan Omidshafiei, Dong-Ki Kim, Abhishek Gupta, and David Fridovich-Keil
Link to paper and project website.
