# No More Blind Spots: Learning Vision-Based Omnidirectional Bipedal Locomotion for Challenging Terrain

arXiv: https://arxiv.org/abs/2508.11929 · HTML: https://mlsanity.com/paper/2508.11929/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 33 references that are in the map.
- First posted on arXiv 16 Aug 2025.

## Builds on (most cited first, 28 of 28)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst](https://mlsanity.com/paper/1812.03079/) (2018)
- [Learning by Cheating](https://mlsanity.com/paper/1912.12294/) (2019)
- [Learning robust perceptive locomotion for quadrupedal robots in the wild](https://mlsanity.com/paper/2201.08117/) (2022)
- [Asymmetric Actor Critic for Image-Based Robot Learning](https://mlsanity.com/paper/1710.06542/) (2017)
- [A System for General In-Hand Object Re-Orientation](https://mlsanity.com/paper/2111.03043/) (2021)
- [Extreme Parkour with Legged Robots](https://mlsanity.com/paper/2309.14341/) (2023)
- [Legged Locomotion in Challenging Terrains using Egocentric Vision](https://mlsanity.com/paper/2211.07638/) (2022)
- [Robot Parkour Learning](https://mlsanity.com/paper/2309.05665/) (2023)
- [Humanoid Parkour Learning](https://mlsanity.com/paper/2406.10759/) (2024)
- [Blind Bipedal Stair Traversal via Sim-to-Real Reinforcement Learning](https://mlsanity.com/paper/2105.08328/) (2021)
- [DTC: Deep Tracking Control](https://mlsanity.com/paper/2309.15462/) (2023)
- [Learning to Jump from Pixels](https://mlsanity.com/paper/2110.15344/) (2021)
- [Elevation Mapping for Locomotion and Navigation using GPU](https://mlsanity.com/paper/2204.12876/) (2022)
- [Learning Vision-Based Bipedal Locomotion for Challenging Terrain](https://mlsanity.com/paper/2309.14594/) (2023)
- [Learning to Drive from Simulation without Real World Labels](https://mlsanity.com/paper/1812.03823/) (2018)
- [Learning Humanoid Locomotion over Challenging Terrain](https://mlsanity.com/paper/2410.03654/) (2024)
- [Learning Visual Locomotion with Cross-Modal Supervision](https://mlsanity.com/paper/2211.03785/) (2022)
- [CTS: Concurrent Teacher-Student Reinforcement Learning for Legged Locomotion](https://mlsanity.com/paper/2405.10830/) (2024)
- [Resilient Legged Local Navigation: Learning to Traverse with Compromised Perception End-to-End](https://mlsanity.com/paper/2310.03581/) (2023)
- [Revisiting Reward Design and Evaluation for Robust Humanoid Standing and Walking](https://mlsanity.com/paper/2404.19173/) (2024)
- [TWIST: Teacher-Student World Model Distillation for Efficient Sim-to-Real Transfer](https://mlsanity.com/paper/2311.03622/) (2023)
- [Privileged Sensing Scaffolds Reinforcement Learning](https://mlsanity.com/paper/2405.14853/) (2024)
- [Neural Scene Representation for Locomotion on Structured Terrain](https://mlsanity.com/paper/2206.08077/) (2022)
- [Learning Dynamic Bipedal Walking Across Stepping Stones](https://mlsanity.com/paper/2205.01807/) (2022)
- [Learn to Teach: Sample-Efficient Privileged Learning for Humanoid Locomotion over Diverse Terrains](https://mlsanity.com/paper/2402.06783/) (2024)
- [VMTS: Vision-Assisted Teacher-Student Reinforcement Learning for Multi-Terrain Locomotion in Bipedal Robots](https://mlsanity.com/paper/2503.07049/) (2025)

## Built on by (most cited first, 3 of 3)

- [Architecture Is All You Need: Diversity-Enabled Sweet Spots for Robust Humanoid Locomotion](https://mlsanity.com/paper/2510.14947/) (2025)
- [Global-Local Attention Decomposition for Terrain Encoding in Humanoid Perceptive Locomotion](https://mlsanity.com/paper/2606.00637/) (2026)
- [Learning Terrain Aware Bipedal Locomotion via Reduced Dimensional Perceptual Representations](https://mlsanity.com/paper/2512.12993/) (2025)

## Abstract (arXiv)

Effective bipedal locomotion in dynamic environments, such as cluttered indoor spaces or uneven terrain, requires agile and adaptive movement in all directions. This necessitates omnidirectional terrain sensing and a controller capable of processing such input. We present a learning framework for vision-based omnidirectional bipedal locomotion, enabling seamless movement using depth images. A key challenge is the high computational cost of rendering omnidirectional depth images in simulation, making traditional sim-to-real reinforcement learning (RL) impractical. Our method combines a robust blind controller with a teacher policy that supervises a vision-based student policy, trained on noise-augmented terrain data to avoid rendering costs during RL and ensure robustness. We also introduce a data augmentation technique for supervised student training, accelerating training by up to 10 times compared to conventional methods. Our framework is validated through simulation and real-world tests, demonstrating effective omnidirectional locomotion with minimal reliance on expensive rendering. This is, to the best of our knowledge, the first demonstration of vision-based omnidirectional bipedal locomotion, showcasing its adaptability to diverse terrains.