mlsanityData to 7 Oct 2026

Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning

arXiv 2109.11978 ↗First posted 24 Sep 2021cs.RO cs.LGcited by 395

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

In this work, we present and study a training set-up that achieves fast policy generation for real-world robotic tasks by using massive parallelism on a single workstation GPU. We analyze and discuss the impact of different training algorithm components in the massively parallel regime on the final policy performance and training times. In addition, we present a novel game-inspired curriculum that is well suited for training with thousands of simulated robots in parallel. We evaluate the approach by training the quadrupedal robot ANYmal to walk on challenging terrain. The parallel approach allows training policies for flat terrain in under four minutes, and in twenty minutes for uneven terrain. This represents a speedup of multiple orders of magnitude compared to previous work. Finally, we transfer the policies to the real robot to validate the approach. We open-source our training code to help accelerate further research in the field of learned legged locomotion.

Key facts. Cited by 395 papers in the map (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 20 of its 29 references that are in the map. First posted on arXiv 24 Sep 2021.

Builds on

20 of 29 references.

References · 20

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  3. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  4. 2019Solving Rubik's Cube with a Robot Handcited by 699
  5. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 494
  6. 2019Learning agile and dynamic motor skills for legged robotscited by 402
  7. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
  8. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
  9. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
  10. 2019Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutionscited by 182
  11. 2017Emergence of Locomotion Behaviours in Rich Environmentscited by 159
  12. 2018GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learningcited by 77
8 more
  1. 2017Self-supervised Deep Reinforcement Learning with Generalized Computation Graphs for Robot Navigationcited by 74
  2. 2021Blind Bipedal Stair Traversal via Sim-to-Real Reinforcement Learningcited by 73
  3. 2017Time Limits in Reinforcement Learningcited by 72
  4. 2019DeepGait: Planning and Control of Quadrupedal Gaits using Deep Reinforcement Learningcited by 45
  5. 2020ALLSTEPS: Curriculum-driven Learning of Stepping Stone Skillscited by 44
  6. 2018Accelerated Methods for Deep Reinforcement Learningcited by 40
  7. 2020Autonomous Spot: Long-Range Autonomous Exploration of Extreme Environments with Legged Locomotioncited by 18
  8. 2021Large Batch Simulation for Deep Reinforcement Learningcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 46; 2023: 65; 2024: 81; 2025: 98; 2026: 102.

395 papers.

Sort:

By area:

  1. 2022DayDreamer: World Models for Physical Robot Learningcited by 372
  2. 2022Masked Visual Pre-training for Motor Controlcited by 246
  3. 2023Orbit: A Unified Simulation Framework for Interactive Robot Learning Environmentscited by 239
  4. 2024ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AIcited by 216
  5. 2024OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learningcited by 167
  6. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  7. 2023Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusioncited by 137
  8. 2023Real-World Humanoid Locomotion with Reinforcement Learningcited by 132
  9. 2022Legged Locomotion in Challenging Terrains using Egocentric Visioncited by 131
  10. 2022Rapid Locomotion via Reinforcement Learningcited by 130
  11. 2025BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusioncited by 128
  12. 2023Extreme Parkour with Legged Robotscited by 120
38 more
  1. 2024Expressive Whole-Body Control for Humanoid Robotscited by 117
  2. 2023Robot Parkour Learningcited by 115
  3. 2022Walk These Ways: Tuning Robot Control for Generalization with Multiplicity of Behaviorcited by 113
  4. 2024Learning Human-to-Humanoid Real-Time Whole-Body Teleoperationcited by 112
  5. 2023Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learningcited by 111
  6. 2023Universal Humanoid Motion Representations for Physics-Based Controlcited by 109
  7. 2025ASAP: Aligning Simulation and Real-World Physics for Learning Agile Humanoid Whole-Body Skillscited by 104
  8. 2022Deep Whole-Body Control: Learning a Unified Policy for Manipulation and Locomotioncited by 98
  9. 2025TWIST: Teleoperated Whole-Body Imitation Systemcited by 86
  10. 2025Flow Matching Policy Gradientscited by 86
  11. 2025ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learningcited by 83
  12. 2024DrEureka: Language Model Guided Sim-To-Real Transfercited by 78
  13. 2022A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learningcited by 76
  14. 2022QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatarscited by 74
  15. 2022Adversarial Motion Priors Make Good Substitutes for Complex Reward Functionscited by 68
  16. 2025HOMIE: Humanoid Loco-Manipulation with Isomorphic Exoskeleton Cockpitcited by 65
  17. 2025Robust Autonomy Emerges from Self-Playcited by 64
  18. 2023TorchRL: A data-driven decision-making library for PyTorchcited by 59
  19. 2024Visual Whole-Body Control for Legged Loco-Manipulationcited by 58
  20. 2025Visual Imitation Enables Contextual Humanoid Controlcited by 57
  21. 2024GRUtopia: Dream General Robots in a City at Scalecited by 54
  22. 2024Agile But Safe: Learning Collision-Free High-Speed Legged Locomotioncited by 53
  23. 2022Factory: Fast Contact for Robotic Assemblycited by 49
  24. 2023Prompt a Robot to Walk with Large Language Modelscited by 44
  25. 2022NeRF2Real: Sim2real Transfer of Vision-guided Bipedal Motion Skills using Neural Radiance Fieldscited by 41
  26. 2023Neural Volumetric Memory for Visual Locomotion Controlcited by 40
  27. 2022Advanced Skills by Learning Locomotion and Local Navigation End-to-Endcited by 39
  28. 2024GenNBV: Generalizable Next-Best-View Policy for Active 3D Reconstructioncited by 39
  29. 20251000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilitiescited by 39
  30. 2025KungfuBot: Physics-Based Humanoid Whole-Body Control for Learning Highly-Dynamic Skillscited by 39
  31. 2024Advancing Humanoid Locomotion: Mastering Challenging Terrains with Denoising World Model Learningcited by 38
  32. 2023DTC: Deep Tracking Controlcited by 37
  33. 2022GenLoco: Generalized Locomotion Controllers for Quadrupedal Robotscited by 36
  34. 2024Humanoid-Gym: Reinforcement Learning for Humanoid Robot with Zero-Shot Sim2Real Transfercited by 35
  35. 2024One Policy to Run Them All: an End-to-end Learning Approach to Multi-Embodiment Locomotioncited by 34
  36. 2025Learning Humanoid Standing-up Control across Diverse Posturescited by 34
  37. 2023DribbleBot: Dynamic Legged Manipulation in the Wildcited by 33
  38. 2023Barkour: Benchmarking Animal-level Agility with Quadruped Robotscited by 33

50 most cited of 395 All 395 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2020Learning to Walk in the Real World with Minimal Human Effort
  2. 2025GRoQ-LoCO: Generalist and Robot-agnostic Quadruped Locomotion Control using Offline Datasets
  3. 2023A Real-World Quadrupedal Locomotion Benchmark for Offline Reinforcement Learning
  4. 2025GeCCo -- a Generalist Contact-Conditioned Policy for Loco-Manipulation Skills on Legged Robots ↗
  5. 2026Teaching a Robot Dog New Tricks: Diverse Quadruped Skills via Combined Reinforcement and Imitation Learning with Adversarial Task Selection
  6. 2025LocoMamba: Vision-Driven Locomotion via End-to-End Deep Reinforcement Learning with Mamba
  7. 2025STRIDE: Automating Reward Design, Deep Reinforcement Learning Training and Feedback Optimization in Humanoid Robotics Locomotion
  8. 2025MoE-Loco: Mixture of Experts for Multitask Locomotion