mlsanityData to 14 Oct 2026

Learning to Walk from Three Minutes of Real-World Data with Semi-structured Dynamics Models

arXiv 2410.09163 ↗First posted 11 Oct 2024cs.RO cs.LG math.OCcited by 16

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Traditionally, model-based reinforcement learning (MBRL) methods exploit neural networks as flexible function approximators to represent textita priori unknown environment dynamics. However, training data are typically scarce in practice, and these black-box models often fail to generalize. Modeling architectures that leverage known physics can substantially reduce the complexity of system-identification, but break down in the face of complex phenomena such as contact. We introduce a novel framework for learning semi-structured dynamics models for contact-rich systems which seamlessly integrates structured first principles modeling techniques with black-box auto-regressive models. Specifically, we develop an ensemble of probabilistic models to estimate external forces, conditioned on historical observations and actions, and integrate these predictions using known Lagrangian dynamics. With this semi-structured approach, we can make accurate long-horizon predictions with substantially less data than prior methods. We leverage this capability and propose Semi-Structured Reinforcement Learning (textttSSRL) a simple model-based learning framework which pushes the sample complexity boundary for real-world learning. We validate our approach on a real-world Unitree Go1 quadruped robot, learning dynamic gaits -- from scratch -- on both hard and soft surfaces with just a few minutes of real-world data. Video and code are available at: https://sites.google.com/utexas.edu/ssrl

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 32 of its 53 references that are in the map. First posted on arXiv 11 Oct 2024.

Builds on

32 of 53 references.

References · 32

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  3. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  4. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  5. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 604
  6. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  7. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 508
  8. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
  9. 2020Lagrangian Neural Networkscited by 398
  10. 2022Learning robust perceptive locomotion for quadrupedal robots in the wildcited by 382
  11. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 370
  12. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
20 more
  1. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  2. 2019Deep Lagrangian Networks: Using Physics as Model Prior for Deep Learningcited by 212
  3. 2016Value Iteration Networkscited by 160
  4. 2017Imagination-Augmented Agents for Deep Reinforcement Learningcited by 140
  5. 2021Reinforcement Learning for Robust Parameterized Locomotion Control of Bipedal Robotscited by 116
  6. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 114
  7. 2022A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learningcited by 83
  8. 2021Legged Robots that Keep on Learning: Fine-Tuning Locomotion Policies in the Real Worldcited by 71
  9. 2019Data Efficient Reinforcement Learning for Legged Robotscited by 63
  10. 2020RLOC: Terrain-Aware Legged Locomotion using Reinforcement Learning and Optimal Controlcited by 59
  11. 2022Global Planning for Contact-Rich Manipulation via Local Smoothing of Quasi-dynamic Contact Modelscited by 49
  12. 2023Neural Volumetric Memory for Visual Locomotion Controlcited by 44
  13. 2021Neural Networks with Physics-Informed Architectures and Constraints for Dynamical Systems Modelingcited by 41
  14. 2020ContactNets: Learning Discontinuous Contact Dynamics with Smooth, Implicit Representationscited by 38
  15. 2022Safe Reinforcement Learning for Legged Locomotioncited by 35
  16. 2022CPG-RL: Learning Central Pattern Generators for Quadruped Locomotioncited by 34
  17. 2022Lyapunov Design for Robust and Efficient Robotic Reinforcement Learningcited by 28
  18. 2021SimGAN: Hybrid Simulator Identification for Domain Adaptation via Adversarial Reinforcement Learningcited by 27
  19. 2021Bundled Gradients through Contact via Randomized Smoothingcited by 24
  20. 2023Smoothed Online Learning for Prediction in Piecewise Affine Systemscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 6.

16 papers.

Sort:

By area:

  1. 2025Robotic World Model: A Neural Network Simulator for Robust Policy Optimization in Roboticscited by 51
  2. 2025Uncertainty-Aware Robotic World Model Makes Offline Model-Based Reinforcement Learning Work on Real Robotscited by 17
  3. 2025Towards Embodiment Scaling Laws in Robot Locomotioncited by 16
  4. 2025Humanoid Whole-Body Locomotion on Narrow Terrain via Dynamic Balance and Reinforcement Learningcited by 11
  5. 2025Gait in Eight: Efficient On-Robot Learning for Omnidirectional Quadruped Locomotioncited by 8
  6. 2026Simulation Distillation: Pretraining World Models in Simulation for Rapid Real-World Adaptationcited by 6
  7. 2025Symmetry-Guided Memory Augmentation for Efficient Locomotion Learningcited by 3
  8. 2025Meta-Learning Online Dynamics Model Adaptation in Off-Road Autonomous Drivingcited by 3
  9. 2026PH-Dreamer: A Physics-Driven World Model via Port-Hamiltonian Generative Dynamicscited by 3
  10. 2025Learning to Walk With Less: A Dyna-Style Approach to Quadrupedal Locomotioncited by 2
  11. 2025Learning Soft Robotic Dynamics with Active Explorationcited by 2
  12. 2026Shape Your Body: Value Gradients for Multi-Embodiment Robot Designcited by 2
4 more
  1. 2026Towards Bridging the Gap between Large-Scale Pretraining and Efficient Finetuning for Humanoid Controlcited by 1
  2. 2026ECO: Energy-Constrained Optimization with Reinforcement Learning for Humanoid Walkingcited by 1
  3. 2025Data-Driven Modeling and Correction of Vehicle Dynamicscited by 0
  4. 2026CTS-MoE: Implicit Terrain Adaptation via Mixture-of-Experts for Perceptive Locomotioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Accelerating Model-Based Reinforcement Learning with State-Space World Models
  2. 2020Differentiable Physics Models for Real-world Offline Model-based Reinforcement Learning
  3. 2024Off-dynamics Conditional Diffusion Planners
  4. 2026Dynamics Are Learned, Not Told: Semi-Supervised Discovery of Latent Dynamics Geometries For Zero-Shot Policy Adaptation
  5. 2025Composite Flow Matching for Reinforcement Learning with Shifted-Dynamics Data
  6. 2024PWM: Policy Learning with Multi-Task World Models
  7. 2021Dream to Explore: Adaptive Simulations for Autonomous Systems
  8. 2024The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning