# Grow Your Limits: Continuous Improvement with Real-World RL for Robotic Locomotion

arXiv: https://arxiv.org/abs/2310.17634 · HTML: https://mlsanity.com/paper/2310.17634/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 64 references that are in the map.
- First posted on arXiv 26 Oct 2023.

## Builds on (most cited first, 30 of 40)

- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning](https://mlsanity.com/paper/2109.11978/) (2021)
- [RMA: Rapid Motor Adaptation for Legged Robots](https://mlsanity.com/paper/2107.04034/) (2021)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [DayDreamer: World Models for Physical Robot Learning](https://mlsanity.com/paper/2206.14176/) (2022)
- [Sim-to-Real: Learning Agile Locomotion For Quadruped Robots](https://mlsanity.com/paper/1804.10332/) (2018)
- [Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection](https://mlsanity.com/paper/1603.02199/) (2016)
- [Randomized Ensembled Double Q-Learning: Learning Fast Without a Model](https://mlsanity.com/paper/2101.05982/) (2021)
- [Deep Visual Foresight for Planning Robot Motion](https://mlsanity.com/paper/1610.00696/) (2016)
- [Learning Agile Robotic Locomotion Skills by Imitating Animals](https://mlsanity.com/paper/2004.00784/) (2020)
- [Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasets](https://mlsanity.com/paper/2109.13396/) (2021)
- [The Primacy Bias in Deep Reinforcement Learning](https://mlsanity.com/paper/2205.07802/) (2022)
- [MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale](https://mlsanity.com/paper/2104.08212/) (2021)
- [How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned](https://mlsanity.com/paper/2102.02915/) (2021)
- [Deep Dynamics Models for Learning Dexterous Manipulation](https://mlsanity.com/paper/1909.11652/) (2019)
- [Rapid Locomotion via Reinforcement Learning](https://mlsanity.com/paper/2205.02824/) (2022)
- [SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning](https://mlsanity.com/paper/2007.04938/) (2020)
- [Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning](https://mlsanity.com/paper/2304.13653/) (2023)
- [The Ingredients of Real-World Robotic Reinforcement Learning](https://mlsanity.com/paper/2004.12570/) (2020)
- [Dropout Q-Functions for Doubly Efficient Reinforcement Learning](https://mlsanity.com/paper/2110.02034/) (2021)
- [Learning to Walk in the Real World with Minimal Human Effort](https://mlsanity.com/paper/2002.08550/) (2020)
- [Model-Based Active Exploration](https://mlsanity.com/paper/1810.12162/) (2018)
- [Minimizing Energy Consumption Leads to the Emergence of Gaits in Legged Robots](https://mlsanity.com/paper/2111.01674/) (2021)
- [Data Efficient Reinforcement Learning for Legged Robots](https://mlsanity.com/paper/1907.03613/) (2019)
- [Sim-to-Real Transfer for Biped Locomotion](https://mlsanity.com/paper/1903.01390/) (2019)
- [Fast and Efficient Locomotion via Learned Gait Transitions](https://mlsanity.com/paper/2104.04644/) (2021)

## Built on by (most cited first, 24 of 24)

- [A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learning](https://mlsanity.com/paper/2509.15937/) (2025)
- [One Policy to Run Them All: an End-to-end Learning Approach to Multi-Embodiment Locomotion](https://mlsanity.com/paper/2409.06366/) (2024)
- [Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimization](https://mlsanity.com/paper/2311.03351/) (2023)
- [A Comprehensive Survey of Cross-Domain Policy Transfer for Embodied Agents](https://mlsanity.com/paper/2402.04580/) (2024)
- [Towards Embodiment Scaling Laws in Robot Locomotion](https://mlsanity.com/paper/2505.05753/) (2025)
- [Gait in Eight: Efficient On-Robot Learning for Omnidirectional Quadruped Locomotion](https://mlsanity.com/paper/2503.08375/) (2025)
- [Multistep Quasimetric Learning for Scalable Goal-conditioned Reinforcement Learning](https://mlsanity.com/paper/2511.07730/) (2025)
- [Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution](https://mlsanity.com/paper/2404.04253/) (2024)
- [Robot Trains Robot: Automatic Real-World Policy Adaptation and Learning for Humanoids](https://mlsanity.com/paper/2508.12252/) (2025)
- [Conformal Policy Learning for Sensorimotor Control Under Distribution Shifts](https://mlsanity.com/paper/2311.01457/) (2023)
- [Simulation Distillation: Pretraining World Models in Simulation for Rapid Real-World Adaptation](https://mlsanity.com/paper/2603.15759/) (2026)
- [LoopSR: Looping Sim-and-Real for Lifelong Policy Adaptation of Legged Robots](https://mlsanity.com/paper/2409.17992/) (2024)
- [Optimizing Quantum Circuits via ZX Diagrams using Reinforcement Learning and Graph Neural Networks](https://mlsanity.com/paper/2504.03429/) (2025)
- [Robust Localization, Mapping, and Navigation for Quadruped Robots](https://mlsanity.com/paper/2505.02272/) (2025)
- [Shape Your Body: Value Gradients for Multi-Embodiment Robot Design](https://mlsanity.com/paper/2606.00702/) (2026)
- [Efficient Sparse-Reward Goal-Conditioned Reinforcement Learning with a High Replay Ratio and Regularization](https://mlsanity.com/paper/2312.05787/) (2023)
- [TIMRL: A Novel Meta-Reinforcement Learning Framework for Non-Stationary and Multi-Task Environments](https://mlsanity.com/paper/2501.07146/) (2025)
- [Real-Time Reinforcement Learning for Dynamic Tasks with a Parallel Soft Robot](https://mlsanity.com/paper/2509.19525/) (2025)
- [SLowRL: Safe Low-Rank Adaptation Reinforcement Learning for Locomotion](https://mlsanity.com/paper/2603.17092/) (2026)
- [BricksRL: A Platform for Democratizing Robotics and Reinforcement Learning Research and Education with LEGO](https://mlsanity.com/paper/2406.17490/) (2024)
- [Dynamic Obstacle Avoidance with Bounded Rationality Adversarial Reinforcement Learning](https://mlsanity.com/paper/2503.11467/) (2025)
- [Expanding-and-Shrinking Binary Neural Networks](https://mlsanity.com/paper/2503.23709/) (2025)
- [SafeExplorer: An Unbiased Policy Gradient for Reinforcement Learning with Recovery Interventions](https://mlsanity.com/paper/2607.08925/) (2026)
- [HELP: Human-Efficient Large-Scale Robot Post-Training with Rollout Segmentation](https://mlsanity.com/paper/2607.09776/) (2026)

## Abstract (arXiv)

Deep reinforcement learning (RL) can enable robots to autonomously acquire complex behaviors, such as legged locomotion. However, RL in the real world is complicated by constraints on efficiency, safety, and overall training stability, which limits its practical applicability. We present APRL, a policy regularization framework that modulates the robot's exploration over the course of training, striking a balance between flexible improvement potential and focused, efficient exploration. APRL enables a quadrupedal robot to efficiently learn to walk entirely in the real world within minutes and continue to improve with more training where prior work saturates in performance. We demonstrate that continued training with APRL results in a policy that is substantially more capable of navigating challenging situations and is able to adapt to changes in dynamics with continued training.