# Learning to Walk from Three Minutes of Real-World Data with Semi-structured Dynamics Models

arXiv: https://arxiv.org/abs/2410.09163 · HTML: https://mlsanity.com/paper/2410.09163/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 32 of its 53 references that are in the map.
- First posted on arXiv 11 Oct 2024.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 30 of 32)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [TD-MPC2: Scalable, Robust World Models for Continuous Control](https://mlsanity.com/paper/2310.16828/) (2023)
- [RMA: Rapid Motor Adaptation for Legged Robots](https://mlsanity.com/paper/2107.04034/) (2021)
- [Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation](https://mlsanity.com/paper/2106.13281/) (2021)
- [Lagrangian Neural Networks](https://mlsanity.com/paper/2003.04630/) (2020)
- [Learning robust perceptive locomotion for quadrupedal robots in the wild](https://mlsanity.com/paper/2201.08117/) (2022)
- [COMBO: Conservative Offline Model-Based Policy Optimization](https://mlsanity.com/paper/2102.08363/) (2021)
- [Randomized Ensembled Double Q-Learning: Learning Fast Without a Model](https://mlsanity.com/paper/2101.05982/) (2021)
- [The Primacy Bias in Deep Reinforcement Learning](https://mlsanity.com/paper/2205.07802/) (2022)
- [Deep Lagrangian Networks: Using Physics as Model Prior for Deep Learning](https://mlsanity.com/paper/1907.04490/) (2019)
- [Value Iteration Networks](https://mlsanity.com/paper/1602.02867/) (2016)
- [Imagination-Augmented Agents for Deep Reinforcement Learning](https://mlsanity.com/paper/1707.06203/) (2017)
- [Reinforcement Learning for Robust Parameterized Locomotion Control of Bipedal Robots](https://mlsanity.com/paper/2103.14295/) (2021)
- [Dropout Q-Functions for Doubly Efficient Reinforcement Learning](https://mlsanity.com/paper/2110.02034/) (2021)
- [A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learning](https://mlsanity.com/paper/2208.07860/) (2022)
- [Legged Robots that Keep on Learning: Fine-Tuning Locomotion Policies in the Real World](https://mlsanity.com/paper/2110.05457/) (2021)
- [Data Efficient Reinforcement Learning for Legged Robots](https://mlsanity.com/paper/1907.03613/) (2019)
- [RLOC: Terrain-Aware Legged Locomotion using Reinforcement Learning and Optimal Control](https://mlsanity.com/paper/2012.03094/) (2020)
- [Global Planning for Contact-Rich Manipulation via Local Smoothing of Quasi-dynamic Contact Models](https://mlsanity.com/paper/2206.10787/) (2022)
- [Neural Volumetric Memory for Visual Locomotion Control](https://mlsanity.com/paper/2304.01201/) (2023)
- [Neural Networks with Physics-Informed Architectures and Constraints for Dynamical Systems Modeling](https://mlsanity.com/paper/2109.06407/) (2021)
- [ContactNets: Learning Discontinuous Contact Dynamics with Smooth, Implicit Representations](https://mlsanity.com/paper/2009.11193/) (2020)
- [Safe Reinforcement Learning for Legged Locomotion](https://mlsanity.com/paper/2203.02638/) (2022)
- [CPG-RL: Learning Central Pattern Generators for Quadruped Locomotion](https://mlsanity.com/paper/2211.00458/) (2022)
- [Lyapunov Design for Robust and Efficient Robotic Reinforcement Learning](https://mlsanity.com/paper/2208.06721/) (2022)
- [SimGAN: Hybrid Simulator Identification for Domain Adaptation via Adversarial Reinforcement Learning](https://mlsanity.com/paper/2101.06005/) (2021)

## Built on by (most cited first, 16 of 16)

- [Robotic World Model: A Neural Network Simulator for Robust Policy Optimization in Robotics](https://mlsanity.com/paper/2501.10100/) (2025)
- [Uncertainty-Aware Robotic World Model Makes Offline Model-Based Reinforcement Learning Work on Real Robots](https://mlsanity.com/paper/2504.16680/) (2025)
- [Towards Embodiment Scaling Laws in Robot Locomotion](https://mlsanity.com/paper/2505.05753/) (2025)
- [Humanoid Whole-Body Locomotion on Narrow Terrain via Dynamic Balance and Reinforcement Learning](https://mlsanity.com/paper/2502.17219/) (2025)
- [Gait in Eight: Efficient On-Robot Learning for Omnidirectional Quadruped Locomotion](https://mlsanity.com/paper/2503.08375/) (2025)
- [Simulation Distillation: Pretraining World Models in Simulation for Rapid Real-World Adaptation](https://mlsanity.com/paper/2603.15759/) (2026)
- [Symmetry-Guided Memory Augmentation for Efficient Locomotion Learning](https://mlsanity.com/paper/2502.01521/) (2025)
- [Meta-Learning Online Dynamics Model Adaptation in Off-Road Autonomous Driving](https://mlsanity.com/paper/2504.16923/) (2025)
- [PH-Dreamer: A Physics-Driven World Model via Port-Hamiltonian Generative Dynamics](https://mlsanity.com/paper/2605.18303/) (2026)
- [Learning to Walk With Less: A Dyna-Style Approach to Quadrupedal Locomotion](https://mlsanity.com/paper/2509.06296/) (2025)
- [Learning Soft Robotic Dynamics with Active Exploration](https://mlsanity.com/paper/2510.27428/) (2025)
- [Shape Your Body: Value Gradients for Multi-Embodiment Robot Design](https://mlsanity.com/paper/2606.00702/) (2026)
- [Towards Bridging the Gap between Large-Scale Pretraining and Efficient Finetuning for Humanoid Control](https://mlsanity.com/paper/2601.21363/) (2026)
- [ECO: Energy-Constrained Optimization with Reinforcement Learning for Humanoid Walking](https://mlsanity.com/paper/2602.06445/) (2026)
- [Data-Driven Modeling and Correction of Vehicle Dynamics](https://mlsanity.com/paper/2512.00289/) (2025)
- [CTS-MoE: Implicit Terrain Adaptation via Mixture-of-Experts for Perceptive Locomotion](https://mlsanity.com/paper/2606.19633/) (2026)

## Abstract (arXiv)

Traditionally, model-based reinforcement learning (MBRL) methods exploit neural networks as flexible function approximators to represent textita priori unknown environment dynamics. However, training data are typically scarce in practice, and these black-box models often fail to generalize. Modeling architectures that leverage known physics can substantially reduce the complexity of system-identification, but break down in the face of complex phenomena such as contact. We introduce a novel framework for learning semi-structured dynamics models for contact-rich systems which seamlessly integrates structured first principles modeling techniques with black-box auto-regressive models. Specifically, we develop an ensemble of probabilistic models to estimate external forces, conditioned on historical observations and actions, and integrate these predictions using known Lagrangian dynamics. With this semi-structured approach, we can make accurate long-horizon predictions with substantially less data than prior methods. We leverage this capability and propose Semi-Structured Reinforcement Learning (textttSSRL) a simple model-based learning framework which pushes the sample complexity boundary for real-world learning. We validate our approach on a real-world Unitree Go1 quadruped robot, learning dynamic gaits -- from scratch -- on both hard and soft surfaces with just a few minutes of real-world data. Video and code are available at: https://sites.google.com/utexas.edu/ssrl