# Distillation-PPO: A Novel Two-Stage Reinforcement Learning Framework for Humanoid Robot Perceptive Locomotion

arXiv: https://arxiv.org/abs/2503.08299 · HTML: https://mlsanity.com/paper/2503.08299/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 28 of its 31 references that are in the map.
- First posted on arXiv 11 Mar 2025.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 28 of 28)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning](https://mlsanity.com/paper/2109.11978/) (2021)
- [RMA: Rapid Motor Adaptation for Legged Robots](https://mlsanity.com/paper/2107.04034/) (2021)
- [FAST-LIO2: Fast Direct LiDAR-inertial Odometry](https://mlsanity.com/paper/2107.06829/) (2021)
- [Extreme Parkour with Legged Robots](https://mlsanity.com/paper/2309.14341/) (2023)
- [ANYmal Parkour: Learning Agile Navigation for Quadrupedal Robots](https://mlsanity.com/paper/2306.14874/) (2023)
- [Robot Parkour Learning](https://mlsanity.com/paper/2309.05665/) (2023)
- [Reinforcement Learning for Versatile, Dynamic, and Robust Bipedal Locomotion Control](https://mlsanity.com/paper/2401.16889/) (2024)
- [Sim-to-Real Learning of All Common Bipedal Gaits via Periodic Reward Composition](https://mlsanity.com/paper/2011.01387/) (2020)
- [Blind Bipedal Stair Traversal via Sim-to-Real Reinforcement Learning](https://mlsanity.com/paper/2105.08328/) (2021)
- [Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformers](https://mlsanity.com/paper/2107.03996/) (2021)
- [DTC: Deep Tracking Control](https://mlsanity.com/paper/2309.15462/) (2023)
- [Neural Volumetric Memory for Visual Locomotion Control](https://mlsanity.com/paper/2304.01201/) (2023)
- [Elevation Mapping for Locomotion and Navigation using GPU](https://mlsanity.com/paper/2204.12876/) (2022)
- [Learning Vision-Based Bipedal Locomotion for Challenging Terrain](https://mlsanity.com/paper/2309.14594/) (2023)
- [Robust and Versatile Bipedal Jumping Control through Reinforcement Learning](https://mlsanity.com/paper/2302.09450/) (2023)
- [Whole-body Humanoid Robot Locomotion with Human Reference](https://mlsanity.com/paper/2402.18294/) (2024)
- [Learning Memory-Based Control for Human-Scale Bipedal Locomotion](https://mlsanity.com/paper/2006.02402/) (2020)
- [Vision-Guided Quadrupedal Locomotion in the Wild with Multi-Modal Delay Randomization](https://mlsanity.com/paper/2109.14549/) (2021)
- [Resilient Legged Local Navigation: Learning to Traverse with Compromised Perception End-to-End](https://mlsanity.com/paper/2310.03581/) (2023)
- [Feedback Control of a Cassie Bipedal Robot: Walking, Standing, and Riding a Segway](https://mlsanity.com/paper/1809.07279/) (2018)
- [Learning Bipedal Walking for Humanoids with Current Feedback](https://mlsanity.com/paper/2303.03724/) (2023)
- [Benchmarking Potential Based Rewards for Learning Humanoid Locomotion](https://mlsanity.com/paper/2307.10142/) (2023)
- [Tailoring Solution Accuracy for Fast Whole-body Model Predictive Control of Legged Robots](https://mlsanity.com/paper/2407.10789/) (2024)
- [Neural Scene Representation for Locomotion on Structured Terrain](https://mlsanity.com/paper/2206.08077/) (2022)
- [Decentralized Motor Skill Learning for Complex Robotic Systems](https://mlsanity.com/paper/2306.17411/) (2023)

## Built on by (most cited first, 10 of 10)

- [LOVON: Legged Open-Vocabulary Object Navigator](https://mlsanity.com/paper/2507.06747/) (2025)
- [DreamPolicy: A Unified World-model Policy for Scalable Humanoid Locomotion](https://mlsanity.com/paper/2505.18780/) (2025)
- [Humanoid Occupancy: Enabling A Generalized Multimodal Occupancy Perception System on Humanoid Robots](https://mlsanity.com/paper/2507.20217/) (2025)
- [FT-NCFM: An Influence-Aware Data Distillation Framework for Efficient VLA Models](https://mlsanity.com/paper/2511.16233/) (2025)
- [AnyBody: Free-Form Whole-Body Humanoid Control from Arbitrary Keypoint Guidance](https://mlsanity.com/paper/2606.29209/) (2026)
- [Pretraining in Actor-Critic Reinforcement Learning for Locomotion](https://mlsanity.com/paper/2510.12363/) (2025)
- [Learning Terrain Aware Bipedal Locomotion via Reduced Dimensional Perceptual Representations](https://mlsanity.com/paper/2512.12993/) (2025)
- [MoASE++: Mixture of Activation Sparsity Experts with Domain-Adaptive On-policy Distillation for Continual Test Time Adaptation](https://mlsanity.com/paper/2605.17743/) (2026)
- [STRIDER: Stepping-Enabled Multi-Gait Hierarchical 3D Loco-Manipulation Framework for Humanoid Robots](https://mlsanity.com/paper/2609.23483/) (2026)
- [From Video to Vision: Learning Generalizable Humanoid Carrying from Human Motion](https://mlsanity.com/paper/2610.14561/) (2026)

## Abstract (arXiv)

In recent years, humanoid robots have garnered significant attention from both academia and industry due to their high adaptability to environments and human-like characteristics. With the rapid advancement of reinforcement learning, substantial progress has been made in the walking control of humanoid robots. However, existing methods still face challenges when dealing with complex environments and irregular terrains. In the field of perceptive locomotion, existing approaches are generally divided into two-stage methods and end-to-end methods. Two-stage methods first train a teacher policy in a simulated environment and then use distillation techniques, such as DAgger, to transfer the privileged information learned as latent features or actions to the student policy. End-to-end methods, on the other hand, forgo the learning of privileged information and directly learn policies from a partially observable Markov decision process (POMDP) through reinforcement learning. However, due to the lack of supervision from a teacher policy, end-to-end methods often face difficulties in training and exhibit unstable performance in real-world applications. This paper proposes an innovative two-stage perceptive locomotion framework that combines the advantages of teacher policies learned in a fully observable Markov decision process (MDP) to regularize and supervise the student policy. At the same time, it leverages the characteristics of reinforcement learning to ensure that the student policy can continue to learn in a POMDP, thereby enhancing the model's upper bound. Our experimental results demonstrate that our two-stage training framework achieves higher training efficiency and stability in simulated environments, while also exhibiting better robustness and generalization capabilities in real-world applications.