# Learning Visual Parkour from Generated Images

arXiv: https://arxiv.org/abs/2411.00083 · HTML: https://mlsanity.com/paper/2411.00083/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Oct 2024.

## Built on by (most cited first, 18 of 18)

- [Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion](https://mlsanity.com/paper/2506.08009/) (2025)
- [Residual Off-Policy RL for Finetuning Behavior Cloning Policies](https://mlsanity.com/paper/2509.19301/) (2025)
- [VR-Robo: A Real-to-Sim-to-Real Framework for Visual Robot Navigation and Locomotion](https://mlsanity.com/paper/2502.01536/) (2025)
- [GaussGym: An open-source real-to-sim framework for learning locomotion from pixels](https://mlsanity.com/paper/2510.15352/) (2025)
- [Perceptive Humanoid Parkour: Chaining Dynamic Human Skills via Motion Matching](https://mlsanity.com/paper/2602.15827/) (2026)
- [Opening the Sim-to-Real Door for Humanoid Pixel-to-Action Policy Transfer](https://mlsanity.com/paper/2512.01061/) (2025)
- [The Sound of Simulation: Learning Multimodal Sim-to-Real Robot Policies with Generative Audio](https://mlsanity.com/paper/2507.02864/) (2025)
- [Dreamland: Controllable World Creation with Simulator and Generative Models](https://mlsanity.com/paper/2506.08006/) (2025)
- [SLIM: Sim-to-Real Legged Instructive Manipulation via Long-Horizon Visuomotor Learning](https://mlsanity.com/paper/2501.09905/) (2025)
- [Lucid-XR: An Extended-Reality Data Engine for Robotic Manipulation](https://mlsanity.com/paper/2605.00244/) (2026)
- [Tether: Autonomous Functional Play with Correspondence-Driven Trajectory Warping](https://mlsanity.com/paper/2603.03278/) (2026)
- [CObL: Toward Zero-Shot Ordinal Layering without User Prompting](https://mlsanity.com/paper/2508.08498/) (2025)
- [R2S-EGO: Dual-Proxy Refinement for Sparse-Capture Real-to-Sim](https://mlsanity.com/paper/2608.06827/) (2026)
- [ExCellGen: Fast, Controllable, Photorealistic 3D Scene Generation from a Single Real-World Exemplar](https://mlsanity.com/paper/2412.16253/) (2024)
- [ADEPT: Adaptive Diffusion Environment for Policy Transfer Sim-to-Real](https://mlsanity.com/paper/2506.01759/) (2025)
- [Iterative Compositional Data Generation for Robot Control](https://mlsanity.com/paper/2512.10891/) (2025)
- [VideoWeaver: Multimodal Multi-View Video-to-Video Transfer for Embodied Agents](https://mlsanity.com/paper/2603.25420/) (2026)
- [Efficient Test-Time Adaptation For Robot Policies](https://mlsanity.com/paper/2610.20318/) (2026)

## Abstract (arXiv)

Fast and accurate physics simulation is an essential component of robot learning, where robots can explore failure scenarios that are difficult to produce in the real world and learn from unlimited on-policy data. Yet, it remains challenging to incorporate RGB-color perception into the sim-to-real pipeline that matches the real world in its richness and realism. In this work, we train a robot dog in simulation for visual parkour. We propose a way to use generative models to synthesize diverse and physically accurate image sequences of the scene from the robot's ego-centric perspective. We present demonstrations of zero-shot transfer to the RGB-only observations of the real world on a robot equipped with a low-cost, off-the-shelf color camera. website visit https://lucidsim.github.io