# Walking with Terrain Reconstruction: Learning to Traverse Risky Sparse Footholds

arXiv: https://arxiv.org/abs/2409.15692 · HTML: https://mlsanity.com/paper/2409.15692/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Sep 2024.

## Built on by (most cited first, 8 of 8)

- [BeamDojo: Learning Agile Humanoid Locomotion on Sparse Footholds](https://mlsanity.com/paper/2502.10363/) (2025)
- [Perceptive Humanoid Parkour: Chaining Dynamic Human Skills via Motion Matching](https://mlsanity.com/paper/2602.15827/) (2026)
- [LIPM-Guided Reinforcement Learning for Stable and Perceptive Locomotion in Bipedal Robots](https://mlsanity.com/paper/2509.09106/) (2025)
- [VMTS: Vision-Assisted Teacher-Student Reinforcement Learning for Multi-Terrain Locomotion in Bipedal Robots](https://mlsanity.com/paper/2503.07049/) (2025)
- [Jump-Start Reinforcement Learning with Self-Evolving Priors for Extreme Monopedal Locomotion](https://mlsanity.com/paper/2507.01243/) (2025)
- [Global-Local Attention Decomposition for Terrain Encoding in Humanoid Perceptive Locomotion](https://mlsanity.com/paper/2606.00637/) (2026)
- [Mind Your Steps: A General Learning Framework for Accurate Humanoid Foothold Tracking](https://mlsanity.com/paper/2606.08253/) (2026)
- [Robot Squid Game: Quadrupedal Locomotion for Traversing Narrow Tunnels](https://mlsanity.com/paper/2605.13665/) (2026)

## Abstract (arXiv)

Traversing risky terrains with sparse footholds presents significant challenges for legged robots, requiring precise foot placement in safe areas. To acquire comprehensive exteroceptive information, prior studies have employed motion capture systems or mapping techniques to generate heightmap for locomotion policy. However, these approaches require specialized pipelines and often introduce additional noise. While depth images from egocentric vision systems are cost-effective, their limited field of view and sparse information hinder the integration of terrain structure details into implicit features, which are essential for generating precise actions. In this paper, we demonstrate that end-to-end reinforcement learning relying solely on proprioception and depth images is capable of traversing risky terrains with high sparsity and randomness. Our method introduces local terrain reconstruction, leveraging the benefits of clear features and sufficient information from the heightmap, which serves as an intermediary for visual feature extraction and motion generation. This allows the policy to effectively represent and memorize critical terrain information. We deploy the proposed framework on a low-cost quadrupedal robot, achieving agile and adaptive locomotion across various challenging terrains and showcasing outstanding performance in real-world scenarios. Video at: youtu.be/Rj9v5EZsn-M.