mlsanityData to 7 Oct 2026

An Efficient Model-Based Approach on Learning Agile Motor Skills without Reinforcement

arXiv 2403.01962 ↗First posted 4 Mar 2024cs.ROcited by 2

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

Learning-based methods have improved locomotion skills of quadruped robots through deep reinforcement learning. However, the sim-to-real gap and low sample efficiency still limit the skill transfer. To address this issue, we propose an efficient model-based learning framework that combines a world model with a policy network. We train a differentiable world model to predict future states and use it to directly supervise a Variational Autoencoder (VAE)-based policy network to imitate real animal behaviors. This significantly reduces the need for real interaction data and allows for rapid policy updates. We also develop a high-level network to track diverse commands and trajectories. Our simulated results show a tenfold sample efficiency increase compared to reinforcement learning methods such as PPO. In real-world testing, our policy achieves proficient command-following performance with only a two-minute data collection period and generalizes well to new speeds and paths.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 29 of its 39 references that are in the map. First posted on arXiv 4 Mar 2024.

Builds on

29 of 39 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Mastering Diverse Domains through World Models10 Jan
  2. 2022DayDreamer: World Models for Physical Robot Learning28 Jun
  3. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning24 Sep
  4. 2020Mastering Atari with Discrete World Models5 Oct
  5. 2019Dream to Control: Learning Behaviors by Latent Imagination3 Dec
  6. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees10 Jul
  7. 2018Model-Ensemble Trust-Region Policy Optimization28 Feb

Other references · 22

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  5. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 494
  6. 2018Sim-to-Real: Learning Agile Locomotion For Quadruped Robotscited by 277
  7. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  8. 2020Learning Agile Robotic Locomotion Skills by Imitating Animalscited by 229
  9. 2018Model-Based Reinforcement Learning via Meta-Policy Optimizationcited by 99
  10. 2022Adversarial Motion Priors Make Good Substitutes for Complex Reward Functionscited by 68
  11. 2021Legged Robots that Keep on Learning: Fine-Tuning Locomotion Policies in the Real Worldcited by 64
  12. 2020Multi-expert learning of adaptive legged locomotioncited by 62
10 more
  1. 2020Dynamics Randomization Revisited:A Case Study for Quadrupedal Locomotioncited by 54
  2. 2021Fast and Efficient Locomotion via Learned Gait Transitionscited by 50
  3. 2023Neural Categorical Priors for Physics-Based Character Controlcited by 37
  4. 2023Lifelike Agility and Play in Quadrupedal Robots using Reinforcement Learning and Generative Pre-trained Modelscited by 30
  5. 2020Learning a Contact-Adaptive Controller for Robust, Efficient Legged Locomotioncited by 22
  6. 2016An Efficient Optimal Planning and Control Framework For Quadrupedal Locomotioncited by 10
  7. 2021Reinforcement Learning with Evolutionary Trajectory Generator: A General Approach for Quadrupedal Locomotioncited by 10
  8. 2019Simultaneous Contact, Gait and Motion Planning for Robust Multi-Legged Locomotion via Mixed-Integer Convex Optimizationcited by 8
  9. 2023Learning Terrain-Adaptive Locomotion with Agile Behaviors by Imitating Animalscited by 8
  10. 2023Terrain-Aware Quadrupedal Locomotion via Reinforcement Learningcited by 4

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024RobotKeyframing: Learning Locomotion with High-Level Objectives via Mixture of Dense and Sparse Rewardscited by 12
  2. 2025DexNDM: Closing the Reality Gap for Dexterous In-Hand Rotation via Joint-Wise Neural Dynamics Modelcited by 8

Similar, not cited

Close in wording, no citation either way

  1. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning
  2. 2025Learning Diverse Natural Behaviors for Enhancing the Agility of Quadrupedal Robots ↗
  3. 2026Learning Loco-Manipulation From SMPC Demonstrations With Sparse Offline-to-Online RL
  4. 2023A Real-World Quadrupedal Locomotion Benchmark for Offline Reinforcement Learning
  5. 2020Neural Dynamic Policies for End-to-End Sensorimotor Learning
  6. 2025Robotic World Model: A Neural Network Simulator for Robust Policy Optimization in Robotics
  7. 2025World Models as Reference Trajectories for Rapid Motor Adaptation
  8. 2026World-Gymnast: Training Robots with Reinforcement Learning in a World Model