mlsanityData to 7 Oct 2026

Sim and Real: Better Together

arXiv 2110.00445 ↗First posted 1 Oct 2021stat.ML cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Simulation is used extensively in autonomous systems, particularly in robotic manipulation. By far, the most common approach is to train a controller in simulation, and then use it as an initial starting point for the real system. We demonstrate how to learn simultaneously from both simulation and interaction with the real environment. We propose an algorithm for balancing the large number of samples from the high throughput but less accurate simulation and the low-throughput, high-fidelity and costly samples from the real environment. We achieve that by maintaining a replay buffer for each environment the agent interacts with. We analyze such multi-environment interaction theoretically, and provide convergence properties, through a novel theoretical replay buffer analysis. We demonstrate the efficacy of our method on a sim-to-real environment.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 34 of its 53 references that are in the map. First posted on arXiv 1 Oct 2021.

Builds on

34 of 53 references.

References · 34

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  5. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  6. 2015Learning Transferable Features with Deep Adaptation Networkscited by 1,323
  7. 2017Hindsight Experience Replaycited by 1,096
  8. 2015Prioritized Experience Replaycited by 1,070
  9. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  10. 2018Learning Dexterous In-Hand Manipulationcited by 643
  11. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  12. 2016Domain Separation Networkscited by 317
22 more
  1. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 312
  2. 2016Learning from Simulated and Unsupervised Images through Adversarial Trainingcited by 287
  3. 2017Learning to Discover Cross-Domain Relations with Generative Adversarial Networkscited by 278
  4. 2018Distributed Prioritized Experience Replaycited by 260
  5. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
  6. 2016CAD2RL: Real Single-Image Flight without a Single Real Imagecited by 227
  7. 2018A DIRT-T Approach to Unsupervised Domain Adaptationcited by 215
  8. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  9. 2018Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experiencecited by 203
  10. 2018Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networkscited by 171
  11. 2020Revisiting Fundamentals of Experience Replaycited by 171
  12. 2020A Finite Time Analysis of Two Time-Scale Actor Critic Methodscited by 120
  13. 2017A Deeper Look at Experience Replaycited by 104
  14. 2019Finite-Sample Analysis for SARSA with Linear Function Approximationcited by 85
  15. 2019BayesSim: adaptive domain randomization via probabilistic inference for robotics simulatorscited by 78
  16. 2017Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Taskcited by 66
  17. 2019Experience Replay Optimizationcited by 47
  18. 2016Collective Robot Reinforcement Learning with Distributed Asynchronous Guided Policy Searchcited by 25
  19. 2019Generalization through Simulation: Integrating Simulated and Real Data into Deep Reinforcement Learning for Vision-Based Autonomous Flightcited by 25
  20. 2019How to pick the domain randomization parameters for sim-to-real transfer of reinforcement learning policies?cited by 25
  21. 2017The Effects of Memory Replay in Reinforcement Learningcited by 22
  22. 2018Organizing Experience: A Deeper Look at Replay Mechanisms for Sample-based Planning in Continuous State Domainscited by 19

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 4; 2024: 1.

6 papers.

  1. 2022Analysis of Stochastic Processes through Replay Bufferscited by 9
  2. 2023Sim-and-Real Reinforcement Learning for Manipulation: A Consensus-based Approachcited by 2
  3. 2023Hindsight States: Blending Sim and Real Task Elements for Efficient Reinforcement Learningcited by 1
  4. 2023Finite-Time Analysis of Temporal Difference Learning with Experience Replaycited by 1
  5. 2024Which Experiences Are Influential for RL Agents? Efficiently Estimating The Influence of Experiencescited by 1
  6. 2023Which Experiences Are Influential for Your Agent? Policy Iteration with Turn-over Dropoutcited by 0