mlsanityData to 14 Oct 2026

Legged Locomotion in Challenging Terrains using Egocentric Vision

arXiv 2211.07638 ↗First posted 14 Nov 2022cs.RO cs.AI cs.CVcited by 151

Not on the map. Why?

Cited by 52 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Animals are capable of precise and agile locomotion using vision. Replicating this ability has been a long-standing goal in robotics. The traditional approach has been to decompose this problem into elevation mapping and foothold planning phases. The elevation mapping, however, is susceptible to failure and large noise artifacts, requires specialized hardware, and is biologically implausible. In this paper, we present the first end-to-end locomotion system capable of traversing stairs, curbs, stepping stones, and gaps. We show this result on a medium-sized quadruped robot using a single front-facing depth camera. The small size of the robot necessitates discovering specialized gait patterns not seen elsewhere. The egocentric camera requires the policy to remember past information to estimate the terrain under its hind feet. We train our policy in simulation. Training has two phases - first, we train a policy using reinforcement learning with a cheap-to-compute variant of depth image and then in phase 2 distill it into the final policy that uses depth using supervised learning. The resulting policy transfers to the real world and is able to run in real-time on the limited compute of the robot. It can traverse a large variety of terrain while being robust to perturbations like pushes, slippery surfaces, and rocky terrain. Videos are at https://vision-locomotion.github.io

Key facts. Cited by 151 papers in the map (data through 14 Oct 2026). Builds on 42 of its 92 references that are in the map. First posted on arXiv 14 Nov 2022.

Builds on

42 of 92 references.

References · 42

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  5. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  6. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 672
  7. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 604
  8. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 512
  9. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 508
  10. 2019Learning agile and dynamic motor skills for legged robotscited by 504
  11. 2019Learning by Cheatingcited by 383
  12. 2022Learning robust perceptive locomotion for quadrupedal robots in the wildcited by 382
30 more
  1. 2018Sim-to-Real: Learning Agile Locomotion For Quadruped Robotscited by 319
  2. 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 276
  3. 2020Learning Agile Robotic Locomotion Skills by Imitating Animalscited by 251
  4. 2017Computer Vision for Autonomous Vehicles: Problems, Datasets and State of the Artcited by 182
  5. 2017Preparing for the Unknown: Learning a Universal Policy with Online System Identificationcited by 141
  6. 2022In-Hand Object Rotation via Rapid Motor Adaptationcited by 118
  7. 2021Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformerscited by 79
  8. 2021Minimizing Energy Consumption Leads to the Emergence of Gaits in Legged Robotscited by 76
  9. 2021Legged Robots that Keep on Learning: Fine-Tuning Locomotion Policies in the Real Worldcited by 71
  10. 2020Dynamics Randomization Revisited:A Case Study for Quadrupedal Locomotioncited by 59
  11. 2019Sim-to-Real Transfer for Biped Locomotioncited by 56
  12. 2020ALLSTEPS: Curriculum-driven Learning of Stepping Stone Skillscited by 54
  13. 2021Coupling Vision and Proprioception for Navigation of Legged Robotscited by 51
  14. 2019DeepGait: Planning and Control of Quadrupedal Gaits using Deep Reinforcement Learningcited by 50
  15. 2019Learning Fast Adaptation with Meta Strategy Optimizationcited by 48
  16. 2021Learning to Jump from Pixelscited by 45
  17. 2018Policy Transfer with Strategy Optimizationcited by 43
  18. 2020Rapidly Adaptable Legged Robots via Evolutionary Meta-Learningcited by 43
  19. 2022Elevation Mapping for Locomotion and Navigation using GPUcited by 43
  20. 2019Multi-Agent Manipulation via Locomotion using Hierarchical Sim2Realcited by 42
  21. 2019Environment Probing Interaction Policiescited by 39
  22. 2018Deep Drone Racing: Learning Agile Flight in Dynamic Environmentscited by 34
  23. 2021Vision-Guided Quadrupedal Locomotion in the Wild with Multi-Modal Delay Randomizationcited by 23
  24. 2017Learning Ground Traversability from Simulationscited by 20
  25. 2020From Pixels to Legs: Hierarchical Learning of Quadruped Locomotioncited by 20
  26. 2020Zero-Shot Terrain Generalization for Visual Locomotion Policiescited by 13
  27. 2021Vision-aided Dynamic Quadrupedal Locomotion on Discrete Terrain using Motion Librariescited by 13
  28. 2018Fast and Continuous Foothold Adaptation for Dynamic Locomotion through CNNscited by 11
  29. 2019On-line and on-board planning and perception for quadrupedal locomotioncited by 5
  30. 2018Computer Vision for Autonomous Vehiclescited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 38; 2024: 60; 2025: 27; 2026: 26.

151 papers.

Sort:

By area:

  1. 2024DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planningcited by 386
  2. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 327
  3. 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successescited by 176
  4. 2023Extreme Parkour with Legged Robotscited by 164
  5. 2023Robot Parkour Learningcited by 149
  6. 2024Open-TeleVision: Teleoperation with Immersive Active Visual Feedbackcited by 145
  7. 2024Expressive Whole-Body Control for Humanoid Robotscited by 139
  8. 2023Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learningcited by 133
  9. 2023General In-Hand Object Rotation with Vision and Touchcited by 99
  10. 2023LEAP Hand: Low-Cost, Efficient, and Anthropomorphic Hand for Robot Learningcited by 79
  11. 2024SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using Gaussian Splattingcited by 69
  12. 2025Visual Imitation Enables Contextual Humanoid Controlcited by 68
38 more
  1. 2023Imitating Task and Motion Planning with Visuomotor Transformerscited by 66
  2. 2024Visual Whole-Body Control for Legged Loco-Manipulationcited by 62
  3. 2023Multi-Stage Cable Routing through Hierarchical Imitation Learningcited by 60
  4. 2023DTC: Deep Tracking Controlcited by 56
  5. 2024Advancing Humanoid Locomotion: Mastering Challenging Terrains with Denoising World Model Learningcited by 56
  6. 2024Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Taskscited by 52
  7. 2025AMO: Adaptive Motion Optimization for Hyper-Dexterous Humanoid Whole-Body Controlcited by 51
  8. 2024Learning Robust Autonomous Navigation and Locomotion for Wheeled-Legged Robotscited by 48
  9. 2023Not Only Rewards But Also Constraints: Applications on Legged Robot Locomotioncited by 47
  10. 2025BeamDojo: Learning Agile Humanoid Locomotion on Sparse Footholdscited by 46
  11. 2023Neural Volumetric Memory for Visual Locomotion Controlcited by 44
  12. 2023Barkour: Benchmarking Animal-level Agility with Quadruped Robotscited by 41
  13. 2023DribbleBot: Dynamic Legged Manipulation in the Wildcited by 40
  14. 2023Legs as Manipulator: Pushing Quadrupedal Agility Beyond Locomotioncited by 39
  15. 2023Dexterous Functional Graspingcited by 39
  16. 2023Hybrid Internal Model: Learning Agile Legged Locomotion with Simulated Robot Responsecited by 39
  17. 2023Lifelike Agility and Play in Quadrupedal Robots using Reinforcement Learning and Generative Pre-trained Modelscited by 37
  18. 2023Robust and Versatile Bipedal Jumping Control through Reinforcement Learningcited by 36
  19. 2023Learning and Adapting Agile Locomotion Skills by Transferring Experiencecited by 33
  20. 2024Learning Quadruped Locomotion Using Differentiable Simulationcited by 32
  21. 2024Bootstrapping Reinforcement Learning with Imitation for Vision-Based Agile Flightcited by 31
  22. 2024SPIN: Simultaneous Perception, Interaction and Navigationcited by 29
  23. 2024EgoGen: An Egocentric Synthetic Data Generatorcited by 27
  24. 2024Pedipulate: Enabling Manipulation Skills using a Quadruped Robot's Legcited by 27
  25. 2024CTS: Concurrent Teacher-Student Reinforcement Learning for Legged Locomotioncited by 25
  26. 2025The Reality Gap in Robotics: Challenges, Solutions, and Best Practicescited by 24
  27. 2024CaT: Constraints as Terminations for Legged Locomotion Reinforcement Learningcited by 23
  28. 2024Local Policies Enable Zero-shot Long-horizon Manipulationcited by 22
  29. 2024MetaUrban: An Embodied AI Simulation Platform for Urban Micromobilitycited by 21
  30. 2024SAPG: Split and Aggregate Policy Gradientscited by 21
  31. 2023Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Criticcited by 20
  32. 2024Learning Visual Quadrupedal Loco-Manipulation from Demonstrationscited by 20
  33. 2025LocoFormer: Generalist Locomotion via Long-context Adaptationcited by 20
  34. 2023Generalized Animal Imitator: Agile Locomotion with Versatile Motion Priorcited by 19
  35. 2024Neural MP: A Generalist Neural Motion Plannercited by 19
  36. 2025Learning coordinated badminton skills for legged manipulatorscited by 19
  37. 2026Perceptive Humanoid Parkour: Chaining Dynamic Human Skills via Motion Matchingcited by 19
  38. 2024Curriculum-Based Reinforcement Learning for Quadrupedal Jumping: A Reference-free Designcited by 18

50 most cited of 151 All 151 (JSON)