mlsanityData to 7 Oct 2026

Evaluating Real-World Robot Manipulation Policies in Simulation

arXiv 2405.05941 ↗First posted 9 May 2024cs.RO cs.CV cs.LGcited by 484

Not on the map. Why?

Cited by 296 in Vision-language-action models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The field of robotics has made significant advances towards generalist robot manipulation policies. However, real-world evaluation of such policies is not scalable and faces reproducibility challenges, which are likely to worsen as policies broaden the spectrum of tasks they can perform. We identify control and visual disparities between real and simulated environments as key challenges for reliable simulated evaluation and propose approaches for mitigating these gaps without needing to craft full-fidelity digital twins of real-world environments. We then employ these approaches to create SIMPLER, a collection of simulated environments for manipulation policy evaluation on common real robot setups. Through paired sim-and-real evaluations of manipulation policies, we demonstrate strong correlation between policy performance in SIMPLER environments and in the real world. Additionally, we find that SIMPLER evaluations accurately reflect real-world policy behavior modes such as sensitivity to various distribution shifts. We open-source all SIMPLER environments along with our workflow for creating new environments at https://simpler-env.github.io to facilitate research on general-purpose manipulation policies and simulated evaluation frameworks.

Key facts. Cited by 484 papers in the map (data through 7 Oct 2026). Builds on 62 of its 83 references that are in the map. First posted on arXiv 9 May 2024.

Builds on

62 of 83 references.

References · 62

  1. 2023Segment Anythingcited by 8,158
  2. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  4. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  5. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  6. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  7. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  8. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  9. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  10. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  11. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  12. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
38 more
  1. 2022A Generalist Agentcited by 802
  2. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 801
  3. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  4. 2018Learning Dexterous In-Hand Manipulationcited by 643
  5. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  6. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  7. 2023SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Renderingcited by 611
  8. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 584
  9. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  10. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  11. 2023Zero123++: a Single Image to Consistent Multi-view Diffusion Base Modelcited by 516
  12. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  13. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  14. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 430
  15. 2019Learning agile and dynamic motor skills for legged robotscited by 402
  16. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 377
  17. 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 313
  18. 2023UniSim: A Neural Closed-Loop Sensor Simulatorcited by 294
  19. 2023One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusioncited by 283
  20. 2017Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metricscited by 259
  21. 2023Habitat 3.0: A Co-Habitat for Humans, Avatars and Robotscited by 220
  22. 2018Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experiencecited by 203
  23. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 193
  24. 2020RoboTHOR: An Open Simulation-to-Real Embodied AI Platformcited by 190
  25. 2018Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networkscited by 171
  26. 2023TensoIR: Tensorial Inverse Renderingcited by 166
  27. 2021BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environmentscited by 161
  28. 2022Ditto: Building Digital Twins of Articulated Objects from Interactioncited by 152
  29. 2017Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Graspingcited by 138
  30. 2020LiDARsim: Realistic LiDAR Simulation by Leveraging the Real Worldcited by 137
  31. 2022In-Hand Object Rotation via Rapid Motor Adaptationcited by 102
  32. 2021ManipulaTHOR: A Framework for Visual Object Manipulationcited by 101
  33. 2022SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learningcited by 93
  34. 2020RL-CycleGAN: Reinforcement Learning Aware Simulation-To-Realcited by 89
  35. 2023Rotating without Seeing: Towards In-hand Dexterity through Touchcited by 78
  36. 2023Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulationcited by 78
  37. 2020RetinaGAN: An Object-aware Approach to Sim-to-Real Transfercited by 68
  38. 2023ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenescited by 68

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 35; 2025: 177; 2026: 272.

484 papers.

Sort:

By area:

  1. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 502
  2. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 411
  3. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 386
  4. 2024Robotic Control via Embodied Chain-of-Thought Reasoningcited by 340
  5. 2024Latent Action Pretraining from Videoscited by 302
  6. 2025X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Modelcited by 285
  7. 2025LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Modelscited by 278
  8. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  9. 2024ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AIcited by 216
  10. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
  11. 2025MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulationcited by 183
  12. 2024VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Taskscited by 171
38 more
  1. 2024RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulationcited by 166
  2. 2025LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorizationcited by 166
  3. 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 162
  4. 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successescited by 153
  5. 2025ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent Planningcited by 152
  6. 2025Stop Looking for Important Tokens in Multimodal Language Models: Duplication Matters Morecited by 149
  7. 2025Ctrl-World: A Controllable Generative World Model for Robot Manipulationcited by 149
  8. 2024Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformerscited by 145
  9. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 138
  10. 2025mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAscited by 133
  11. 2025Unified Vision-Language-Action Modelcited by 128
  12. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 125
  13. 2025Spatial Forcing: Implicit Spatial Representation Alignment for Vision-language-action Modelcited by 125
  14. 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 117
  15. 2025EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videoscited by 116
  16. 2024GRAPE: Generalizing Robot Policy via Preference Alignmentcited by 106
  17. 2026VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Modelcited by 104
  18. 2026StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developingcited by 103
  19. 2025What Can RL Bring to VLA Generalization? An Empirical Studycited by 102
  20. 2025CAST: Component-Aligned 3D Scene Reconstruction from an RGB Imagecited by 101
  21. 2024Steering Your Generalists: Improving Robotic Foundation Models via Value Guidancecited by 100
  22. 2026A Pragmatic VLA Foundation Modelcited by 97
  23. 2025A Survey on Vision-Language-Action Models: An Action Tokenization Perspectivecited by 96
  24. 2024IRASim: A Fine-Grained World Model for Robot Manipulationcited by 93
  25. 2025Discrete Diffusion VLA: Bringing Discrete Diffusion to Action Decoding in Vision-Language-Action Policiescited by 90
  26. 2025EfficientVLA: Training-Free Acceleration and Compression for Vision-Language-Action Modelscited by 89
  27. 2024What Matters in Building Vision-Language-Action Models for Generalist Robotscited by 88
  28. 2025F1: A Vision-Language-Action Model Bridging Understanding and Generation to Actionscited by 85
  29. 2025VLA-Cache: Efficient Vision-Language-Action Manipulation via Adaptive Token Cachingcited by 83
  30. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  31. 2025SAFE: Multitask Failure Detection for Vision-Language-Action Modelscited by 74
  32. 2025Dita: Scaling Diffusion Transformer for Generalist Vision-Language-Action Policycited by 73
  33. 2025Self-Improving Vision-Language-Action Models with Data Generation via Residual RLcited by 72
  34. 2025InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulationcited by 71
  35. 2024IGOR: Image-GOal Representations are the Atomic Control Units for Foundation Models in Embodied AIcited by 68
  36. 2025Vid2World: Crafting Video Diffusion Models to Interactive World Modelscited by 67
  37. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 66
  38. 2024Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videoscited by 65

50 most cited of 484 All 484 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Robot Policy Evaluation for Sim-to-Real Transfer: A Benchmarking Perspective
  2. 2026X2Real: an eXtensive simulation benchmark for real-world generalist policies ↗
  3. 2025Scalable Policy Evaluation with Video World Models
  4. 2026Grounding Sim-to-Real Generalization in Robotic Manipulation: An Empirical Study with Vision-Language-Action Models
  5. 2026GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluation
  6. 2026ManipArena: Comprehensive Real-world Evaluation of Reasoning-Oriented Generalist Robot Manipulation
  7. 2026Beyond Binary Success: Sample-Efficient and Statistically Rigorous Robot Policy Comparison
  8. 2026RoboWM-Bench: A Benchmark for Evaluating World Models in Robotic Manipulation