# Zero-Shot Terrain Generalization for Visual Locomotion Policies

arXiv: https://arxiv.org/abs/2011.05513 · HTML: https://mlsanity.com/paper/2011.05513/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Nov 2020.

## Built on by (most cited first, 13 of 13)

- [Legged Locomotion in Challenging Terrains using Egocentric Vision](https://mlsanity.com/paper/2211.07638/) (2022)
- [Robot Parkour Learning](https://mlsanity.com/paper/2309.05665/) (2023)
- [Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformers](https://mlsanity.com/paper/2107.03996/) (2021)
- [Vision-Guided Quadrupedal Locomotion in the Wild with Multi-Modal Delay Randomization](https://mlsanity.com/paper/2109.14549/) (2021)
- [Dynamics Generalisation in Reinforcement Learning via Adaptive Context-Aware Policies](https://mlsanity.com/paper/2310.16686/) (2023)
- [Dreaming of Many Worlds: Learning Contextual World Models Aids Zero-Shot Generalization](https://mlsanity.com/paper/2403.10967/) (2024)
- [Inferring Behavior-Specific Context Improves Zero-Shot Generalization in Reinforcement Learning](https://mlsanity.com/paper/2404.09521/) (2024)
- [Learning Coordinated Terrain-Adaptive Locomotion by Imitating a Centroidal Dynamics Planner](https://mlsanity.com/paper/2111.00262/) (2021)
- [Imitation and Adaptation Based on Consistency: A Quadruped Robot Imitates Animals from Videos Using Deep Reinforcement Learning](https://mlsanity.com/paper/2203.05973/) (2022)
- [Multiscale Sensor Fusion and Continuous Control with Neural CDEs](https://mlsanity.com/paper/2203.08715/) (2022)
- [A Transferable Legged Mobile Manipulation Framework Based on Disturbance Predictive Control](https://mlsanity.com/paper/2203.03391/) (2022)
- [Learning Bipedal Walking for Humanoid Robots in Challenging Environments with Obstacle Avoidance](https://mlsanity.com/paper/2410.08212/) (2024)
- [Hierarchical Reinforcement Learning and Value Optimization for Challenging Quadruped Locomotion](https://mlsanity.com/paper/2506.20036/) (2025)

## Abstract (arXiv)

Legged robots have unparalleled mobility on unstructured terrains. However, it remains an open challenge to design locomotion controllers that can operate in a large variety of environments. In this paper, we address this challenge of automatically learning locomotion controllers that can generalize to a diverse collection of terrains often encountered in the real world. We frame this challenge as a multi-task reinforcement learning problem and define each task as a type of terrain that the robot needs to traverse. We propose an end-to-end learning approach that makes direct use of the raw exteroceptive inputs gathered from a simulated 3D LiDAR sensor, thus circumventing the need for ground-truth heightmaps or preprocessing of perception information. As a result, the learned controller demonstrates excellent zero-shot generalization capabilities and can navigate 13 different environments, including stairs, rugged land, cluttered offices, and indoor spaces with humans.