mlsanityData to 7 Oct 2026

Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulation

arXiv 2403.03949 ↗First posted 6 Mar 2024cs.RO cs.AI cs.LGcited by 135

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

Imitation learning methods need significant human supervision to learn policies robust to changes in object poses, physical disturbances, and visual distractors. Reinforcement learning, on the other hand, can explore the environment autonomously to learn robust behaviors but may require impractical amounts of unsafe real-world data collection. To learn performant, robust policies without the burden of unsafe real-world data collection or extensive human supervision, we propose RialTo, a system for robustifying real-world imitation learning policies via reinforcement learning in "digital twin" simulation environments constructed on the fly from small amounts of real-world data. To enable this real-to-sim-to-real pipeline, RialTo proposes an easy-to-use interface for quickly scanning and constructing digital twins of real-world environments. We also introduce a novel "inverse distillation" procedure for bringing real-world demonstrations into simulated environments for efficient fine-tuning, with minimal human intervention and engineering required. We evaluate RialTo across a variety of robotic manipulation problems in the real world, such as robustly stacking dishes on a rack, placing books on a shelf, and six other tasks. RialTo increases (over 67%) in policy robustness without requiring extensive human data collection. Project website and videos at https://real-to-sim-to-real.github.io/RialTo/

Key facts. Cited by 135 papers in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 58 of its 76 references that are in the map. First posted on arXiv 6 Mar 2024.

Builds on

58 of 76 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware23 Apr
  2. 2023NeRF in the Palm of Your Hand: Corrective Augmentation for Robotics via Novel-View Synthesis18 Jan
  3. 2022CACTI: A Framework for Scalable Multi-Task Multi-Scene Visual Imitation Learning12 Dec
  4. 2020RetinaGAN: An Object-aware Approach to Sim-to-Real Transfer6 Nov
  5. 2019Self-Supervised Correspondence in Visuomotor Policy Learning16 Sep

Other references · 53

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  5. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  6. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  7. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  8. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  9. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  10. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  11. 2020Convolutional Occupancy Networkscited by 760
  12. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
38 more
  1. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  2. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  3. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  4. 2023Nerfstudio: A Modular Framework for Neural Radiance Field Developmentcited by 596
  5. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 494
  6. 2021Implicit Behavioral Cloningcited by 442
  7. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 430
  8. 2019Learning agile and dynamic motor skills for legged robotscited by 402
  9. 2019Learning by Cheatingcited by 361
  10. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  11. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  12. 2018Sim-to-Real: Learning Agile Locomotion For Quadruped Robotscited by 277
  13. 2023Orbit: A Unified Simulation Framework for Interactive Robot Learning Environmentscited by 239
  14. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 175
  15. 2021A System for General In-Hand Object Re-Orientationcited by 155
  16. 2022Ditto: Building Digital Twins of Articulated Objects from Interactioncited by 152
  17. 2023Structured World Models from Human Videoscited by 144
  18. 2022DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Realitycited by 136
  19. 2022Rapid Locomotion via Reinforcement Learningcited by 130
  20. 2021Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisationcited by 120
  21. 2023GenSim: Generating Robotic Simulation Tasks via Large Language Modelscited by 120
  22. 2020The Ingredients of Real-World Robotic Reinforcement Learningcited by 111
  23. 2022Visual Dexterity: In-Hand Reorientation of Novel and Complex Object Shapescited by 92
  24. 2021Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Interventioncited by 90
  25. 2020RL-CycleGAN: Reinforcement Learning Aware Simulation-To-Realcited by 89
  26. 2023GOAT: GO to Any Thingcited by 89
  27. 2021Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulationcited by 85
  28. 2018Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Costcited by 79
  29. 2016Interactive Perception: Leveraging Action in Perception and Perception in Actioncited by 77
  30. 2023Reaching the Limit in Autonomous Racing: Optimal Control versus Reinforcement Learningcited by 72
  31. 2023ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenescited by 68
  32. 2019DensePhysNet: Learning Dense Physical Object Representations via Multi-step Dynamic Interactionscited by 61
  33. 2023Robot Fine-Tuning Made Easy: Pre-Training Rewards and Policies for Autonomous Real-World Reinforcement Learningcited by 49
  34. 2024PoCo: Policy Composition from and for Heterogeneous Robot Learningcited by 42
  35. 2022NeRF2Real: Sim2real Transfer of Vision-guided Bipedal Motion Skills using Neural Radiance Fieldscited by 41
  36. 2023Finetuning Offline World Models in the Real Worldcited by 39
  37. 2022DiffCloud: Real-to-Sim from Point Clouds with Differentiable Simulation and Rendering of Deformable Objectscited by 29
  38. 2022Phone2Proc: Bringing Robust Robots Into Our Chaotic Worldcited by 24

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 21; 2025: 62; 2026: 52.

135 papers.

Sort:

By area:

  1. 2024Diffusion Policy Policy Optimizationcited by 255
  2. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  3. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 125
  4. 2025CAST: Component-Aligned 3D Scene Reconstruction from an RGB Imagecited by 101
  5. 2024Articulate-Anything: Automatic Modeling of Articulated Objects via a Vision-Language Foundation Modelcited by 90
  6. 2024What Matters in Building Vision-Language-Action Models for Generalist Robotscited by 88
  7. 2024From Imitation to Refinement -- Residual RL for Precise Assemblycited by 80
  8. 2025ArtGS: Building Interactable Replicas of Complex Articulated Objects via Gaussian Splattingcited by 67
  9. 2024Real2Code: Reconstruct Articulated Objects via Code Generationcited by 64
  10. 2024SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using Gaussian Splattingcited by 63
  11. 2024FLaRe: Achieving Masterful and Adaptive Robot Policies with Large-Scale Reinforcement Learning Fine-Tuningcited by 54
  12. 2025Evaluating Gemini Robotics Policies in a Veo World Simulatorcited by 50
38 more
  1. 2025NavDP: Learning Sim-to-Real Navigation Diffusion Policy with Privileged Information Guidancecited by 46
  2. 2024Dream to Manipulate: Compositional World Models Empowering Robot Imitation Learning with Imaginationcited by 45
  3. 2025ControlVLA: Few-shot Object-centric Adaptation for Pre-trained Vision-Language-Action Modelscited by 43
  4. 2025Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulationcited by 42
  5. 2025Residual Off-Policy RL for Finetuning Behavior Cloning Policiescited by 42
  6. 2025Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoidscited by 40
  7. 2024ActiveSplat: High-Fidelity Scene Reconstruction through Active Gaussian Splattingcited by 32
  8. 2025Digital Twin Catalog: A Large-Scale Photorealistic 3D Object Digital Twin Datasetcited by 29
  9. 2025DRAWER: Digital Reconstruction and Articulation With Environment Realismcited by 29
  10. 2025PolaRiS: Scalable Real-to-Sim Evaluations for Generalist Robot Policiescited by 29
  11. 2025DiWA: Diffusion Policy Adaptation with World Modelscited by 28
  12. 2024ASID: Active Exploration for System Identification in Robotic Manipulationcited by 27
  13. 2025Crossing the Human-Robot Embodiment Gap with Sim-to-Real RL using One Human Demonstrationcited by 27
  14. 2025X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Realcited by 25
  15. 2024Lessons from Learning to Spin "Pens"cited by 22
  16. 2025ReBot: Scaling Robot Learning with Real-to-Sim-to-Real Robotic Video Synthesiscited by 22
  17. 2025GaussGym: An open-source real-to-sim framework for learning locomotion from pixelscited by 22
  18. 2025The Reality Gap in Robotics: Challenges, Solutions, and Best Practicescited by 22
  19. 2025Video2Policy: Scaling up Manipulation Tasks in Simulation through Internet Videoscited by 21
  20. 2025HuB: Learning Extreme Humanoid Balancecited by 21
  21. 2025VR-Robo: A Real-to-Sim-to-Real Framework for Visual Robot Navigation and Locomotioncited by 20
  22. 2025iTACO: Interactable Digital Twins of Articulated Objects from Casually Captured RGBD Videoscited by 18
  23. 2025Robot Learning from a Physical World Modelcited by 18
  24. 2024NeRFs in Robotics: A Surveycited by 16
  25. 2025GSWorld: Closed-Loop Photo-Realistic Simulation Suite for Robotic Manipulationcited by 16
  26. 2025Rapidly Adapting Policies to the Real World via Simulation-Guided Fine-Tuningcited by 15
  27. 2025DEXOP: A Device for Robotic Transfer of Dexterous Human Manipulationcited by 15
  28. 2025VideoArtGS: Building Digital Twins of Articulated Objects from Monocular Videocited by 15
  29. 2025RobotArena ∞: Scalable Robot Benchmarking via Real-to-Sim Translationcited by 15
  30. 2024GRS: Generating Robotic Simulation Tasks from Real-World Imagescited by 13
  31. 2025HoloScene: Simulation-Ready Interactive 3D Worlds from a Single Videocited by 13
  32. 2025Empirical Analysis of Sim-and-Real Cotraining of Diffusion Policies for Planar Pushing from Pixelscited by 12
  33. 2025REALM: A Real-to-Sim Validated Benchmark for Generalization in Robotic Manipulationcited by 12
  34. 2025Real-is-Sim: Bridging the Sim-to-Real Gap with a Dynamic Digital Twincited by 11
  35. 2024Dynamics as Prompts: In-Context Learning for Sim-to-Real System Identificationscited by 10
  36. 2024Robot Learning with Super-Linear Scalingcited by 10
  37. 2025Web2Grasp: Learning Functional Grasps from Web Images of Hand-Object Interactionscited by 10
  38. 2025VT-Refine: Learning Bimanual Assembly with Visuo-Tactile Feedback via Simulation Fine-Tuningcited by 10

50 most cited of 135 All 135 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Robust Learning from Observation with Model Misspecification
  2. 2024Efficient Imitation Learning with Conservative World Models
  3. 2025RoboManipBaselines: A Unified Framework for Imitation Learning in Robotic Manipulation across Real and Simulation Environments
  4. 2024Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning
  5. 2023Get Back Here: Robust Imitation by Return-to-Distribution Planning
  6. 2026WorldSample: Closed-loop Real-robot RL with World Modelling
  7. 2024A context-adaptive policy framework for robust and reactive robotic manipulation via uncertainty-aware imitation learning
  8. 2026Online World Modeling Enables Real-World Inverse Reinforcement Learning from Observation