mlsanityData to 7 Oct 2026

What Matters for Simulation to Online Reinforcement Learning on Real Robots

arXiv 2602.20220 ↗First posted 23 Feb 2026cs.RO cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

We investigate what specific design choices enable successful online reinforcement learning (RL) on physical robots. Across 100 real-world training runs on three distinct robotic platforms, we systematically ablate algorithmic, systems, and experimental decisions that are typically left implicit in prior work. We find that some widely used defaults can be harmful, while a set of robust, readily adopted design choices within standard RL practice yield stable learning across tasks and hardware. These results provide the first large-sample empirical study of such design choices, enabling practitioners to deploy online RL with lower engineering effort.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 39 of its 52 references that are in the map. First posted on arXiv 23 Feb 2026.

Builds on

39 of 52 references.

References · 39

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  8. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  9. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  10. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  11. 2015Variational Dropout and the Local Reparameterization Trickcited by 489
  12. 2019Learning agile and dynamic motor skills for legged robotscited by 402
27 more
  1. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
  2. 2022DayDreamer: World Models for Physical Robot Learningcited by 372
  3. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  4. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  5. 2018Maximum a Posteriori Policy Optimisationcited by 318
  6. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  7. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  8. 2024Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learningcited by 153
  9. 2025BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusioncited by 128
  10. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 104
  11. 2020Learning to Walk in the Real World with Minimal Human Effortcited by 103
  12. 2024Steering Your Generalists: Improving Robotic Foundation Models via Value Guidancecited by 100
  13. 2017Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learningcited by 86
  14. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
  15. 2022A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learningcited by 76
  16. 2021Legged Robots that Keep on Learning: Fine-Tuning Locomotion Policies in the Real Worldcited by 64
  17. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Datacited by 58
  18. 2025MuJoCo Playgroundcited by 57
  19. 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learningcited by 50
  20. 2023IndustReal: Transferring Contact-Rich Assembly Tasks from Simulation to Realitycited by 44
  21. 2023Finetuning Offline World Models in the Real Worldcited by 39
  22. 2021Autonomous Reinforcement Learning: Formalism and Benchmarkingcited by 33
  23. 2021Autonomous Reinforcement Learning via Subgoal Curriculacited by 32
  24. 2025FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Controlcited by 32
  25. 2022A State-Distribution Matching Approach to Non-Episodic Reinforcement Learningcited by 21
  26. 2025Rapidly Adapting Policies to the Real World via Simulation-Guided Fine-Tuningcited by 15
  27. 2023Replay across Experiments: A Natural Extension of Off-Policy RLcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026When Does Non-Uniform Replay Matter in Reinforcement Learning?cited by 0
  2. 2026Beyond One-Size-Fits-All: Diagnosis-Driven Online Reinforcement Learning with Offline Priorscited by 0