# Learning to Walk With Less: A Dyna-Style Approach to Quadrupedal Locomotion

arXiv: https://arxiv.org/abs/2509.06296 · HTML: https://mlsanity.com/paper/2509.06296/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 27 of its 43 references that are in the map.
- First posted on arXiv 8 Sep 2025.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 27 of 27)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning](https://mlsanity.com/paper/2109.11978/) (2021)
- [RMA: Rapid Motor Adaptation for Legged Robots](https://mlsanity.com/paper/2107.04034/) (2021)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [Reinforcement Learning with Augmented Data](https://mlsanity.com/paper/2004.14990/) (2020)
- [Temporal Difference Learning for Model Predictive Control](https://mlsanity.com/paper/2203.04955/) (2022)
- [Learning robust perceptive locomotion for quadrupedal robots in the wild](https://mlsanity.com/paper/2201.08117/) (2022)
- [Rapid Locomotion via Reinforcement Learning](https://mlsanity.com/paper/2205.02824/) (2022)
- [Learning to Walk via Deep Reinforcement Learning](https://mlsanity.com/paper/1812.11103/) (2018)
- [ANYmal Parkour: Learning Agile Navigation for Quadrupedal Robots](https://mlsanity.com/paper/2306.14874/) (2023)
- [Walk These Ways: Tuning Robot Control for Generalization with Multiplicity of Behavior](https://mlsanity.com/paper/2212.03238/) (2022)
- [Data Efficient Reinforcement Learning for Legged Robots](https://mlsanity.com/paper/1907.03613/) (2019)
- [Learning to Combat Compounding-Error in Model-Based Reinforcement Learning](https://mlsanity.com/paper/1912.11206/) (2019)
- [Perceptive Locomotion through Nonlinear Model Predictive Control](https://mlsanity.com/paper/2208.08373/) (2022)
- [Learning Quadruped Locomotion Using Differentiable Simulation](https://mlsanity.com/paper/2403.14864/) (2024)
- [Leveraging Symmetry in RL-based Legged Locomotion Control](https://mlsanity.com/paper/2403.17320/) (2024)
- [Fast Contact-Implicit Model-Predictive Control](https://mlsanity.com/paper/2107.05616/) (2021)
- [Learning to Walk from Three Minutes of Real-World Data with Semi-structured Dynamics Models](https://mlsanity.com/paper/2410.09163/) (2024)
- [Accelerating Model-Based Reinforcement Learning with State-Space World Models](https://mlsanity.com/paper/2502.20168/) (2025)
- [Robust Robot Walker: Learning Agile Locomotion over Tiny Traps](https://mlsanity.com/paper/2409.07409/) (2024)
- [Accelerating Model Predictive Control for Legged Robots through Distributed Optimization](https://mlsanity.com/paper/2403.11742/) (2024)
- [Inverse-Dynamics MPC via Nullspace Resolution](https://mlsanity.com/paper/2209.05375/) (2022)
- [Online Dynamic Trajectory Optimization and Control for a Quadruped Robot](https://mlsanity.com/paper/2008.12687/) (2020)

## Built on by (most cited first, 2 of 2)

- [Learning Terrain-Specialized Policies for Adaptive Locomotion in Challenging Environments](https://mlsanity.com/paper/2509.20635/) (2025)
- [LeCropFollow: Latent Space Planning for Navigation in Unstructured Crop Fields](https://mlsanity.com/paper/2606.31941/) (2026)

## Abstract (arXiv)

Traditional on-policy reinforcement learning (RL) controllers for quadrupedal locomotion often suffer from low data efficiency, requiring millions of interactions with simulated environments to achieve stable control. We integrate model-based techniques that improve sample efficiency by augmenting PPO rollouts with synthetic data in a Dyna-style framework. Our method employs a learned transition model to generate short-horizon synthetic tails for each trajectory, anchored by physics-based simulation to preserve stability. A predefined scheduling strategy gradually integrates synthetic transitions, preventing model usage during early training stages when prediction accuracy is low. Through extensive ablation studies, we analyze how varying data parameters influence PPO's learning behavior. Finally, we validate our method in simulation on a Unitree Go1 robot, reaching convergence with substantially fewer simulation steps (19.64M vs. 27.53M) and a 12.24% reduction in wall-clock training time, without compromising policy performance or convergence. Cross-platform experiments on ANYmal and Unitree Go2 further confirm the framework's ability to learn high-dimensional locomotion control with substantially reduced simulation experience, despite reward trade-offs on complex morphologies.