mlsanityData to 14 Oct 2026

REALM: A Real-to-Sim Validated Benchmark for Generalization in Robotic Manipulation

arXiv 2512.19562 ↗First posted 22 Dec 2025cs.RO cs.AIcited by 15

On the map: Vision-language-action models → Generalist VLA models and benchmarks

Abstract (arXiv)

Vision-Language-Action (VLA) models empower robots to understand and execute tasks described by natural language instructions. However, a key challenge lies in their ability to generalize beyond the specific environments and conditions they were trained on, which is presently difficult and expensive to evaluate in the real-world. To address this gap, we present REALM, a new simulation environment and benchmark designed to evaluate the generalization capabilities of VLA models, with a specific emphasis on establishing a strong correlation between simulated and real-world performance through high-fidelity visuals and aligned robot control. Our environment offers a suite of 15 perturbation factors, 7 manipulation skills, and more than 3,500 objects. Finally, we establish two task sets that form our benchmark and evaluate the π0, π0-FAST, and GR00T N1.5 VLA models, showing that generalization and robustness remain an open challenge. More broadly, we also show that simulation gives us a valuable proxy for the real-world and allows us to systematically probe for and quantify the weaknesses and failure modes of VLAs. Project page: https://martin-sedlacek.com/realm

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Generalist VLA models and benchmarks, one of 13 directions in Vision-language-action models. Builds on 40 of its 52 references that are in the map. First posted on arXiv 22 Dec 2025.

Builds on

40 of 52 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challenges7 May
  2. 2024π0: A Vision-Language-Action Flow Model for General Robot Control31 Oct
  3. 2024OpenVLA: An Open-Source Vision-Language-Action Model13 Jun
  4. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul

Other references · 36

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  6. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,410
  7. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,225
  8. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 852
  9. 2024PaliGemma: A versatile 3B VLM for transfercited by 696
  10. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 665
  11. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 657
  12. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 584
24 more
  1. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 495
  2. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 339
  3. 2024ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AIcited by 229
  4. 2024VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Taskscited by 175
  5. 2025Ctrl-World: A Controllable Generative World Model for Robot Manipulationcited by 167
  6. 2024BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulationcited by 148
  7. 2024Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulationcited by 144
  8. 2024THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulationcited by 133
  9. 2025A Careful Examination of Large Behavior Models for Multitask Dexterous Manipulationcited by 129
  10. 2024Automated Creation of Digital Cousins for Robust Policy Learningcited by 91
  11. 2023ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenescited by 69
  12. 2022VLMbench: A Compositional Benchmark for Vision-and-Language Manipulationcited by 68
  13. 2025RoboArena: Distributed Real-World Evaluation of Generalist Robot Policiescited by 64
  14. 2025RoboVerse: Towards a Unified Platform, Dataset and Benchmark for Scalable and Generalizable Robot Learningcited by 54
  15. 2024Towards Generalizable Vision-Language Robotic Manipulation: A Benchmark and LLM-guided 3D Policycited by 38
  16. 2025PolaRiS: Scalable Real-to-Sim Evaluations for Generalist Robot Policiescited by 33
  17. 2022How Severe is Benchmark-Sensitivity in Video Self-Supervised Learning?cited by 23
  18. 2024Robot Learning as an Empirical Science: Best Practices for Policy Evaluationcited by 23
  19. 2024What Foundation Models can Bring for Robot Learning in Manipulation : A Surveycited by 19
  20. 2025BOP Challenge 2024 on Model-Based and Model-Free 6D Object Pose Estimationcited by 19
  21. 2025RobotArena ∞: Scalable Robot Benchmarking via Real-to-Sim Translationcited by 18
  22. 2025Reliable and Scalable Robot Policy Evaluation with Imperfect Simulatorscited by 14
  23. 2025Guiding Data Collection via Factored Scaling Curvescited by 12
  24. 2024CMA-ES for Safe Optimizationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 15.

15 papers.

Sort:

By area:

  1. 2026RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policiescited by 24
  2. 2026Data Pyramid for Embodied Manipulation: A Surveycited by 6
  3. 2026Drift is a Sampling Error: SNR-Aware Power Distributions for Long-Horizon Robotic Planningcited by 3
  4. 2026Unmasking the Illusion of Embodied Reasoning in Vision-Language-Action Modelscited by 2
  5. 2026You Don't Need To Stay in The Loop: An Agentic Robotics Loop for Robot-Policy Improvementcited by 2
  6. 2026Active Real-World Factor-Based Evaluation for Generalist Robot Policiescited by 1
  7. 2026The Imitator Game: Benchmarking Robot Imitative Ability Beyond Action Predictioncited by 1
  8. 2026Behavior-Skill: A Fine-Grained Benchmark for Evaluating Vision-Language-Action Policies in Long-Horizon Taskscited by 1
  9. 2026RoboSPA: Can VLA Models Go Beyond Simple Scenes and Short-Horizon Tasks?cited by 1
  10. 2026Beyond Binary Success: A Diagnostic Meta-Evaluation Framework for Fine-Grained Manipulationcited by 0
  11. 2026A Practical Recipe Towards Improving Sim-and-Real Correlation for VLA Evaluationcited by 0
  12. 2026Towards Trustworthy Physical Intelligence: From Theory to Practice Across Life Cyclecited by 0
3 more
  1. 2026A Comprehensive Review of Generative Physical Artificial Intelligencecited by 0
  2. 2026Robot-GST: geometry-aware spatial-temporal robot policy representation and evaluationcited by 0
  3. 2026VLA-GAUGE: Systematically Evaluating Generalization and Robustness in Vision-Language-Action Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Colosseum V2: Benchmarking Generalization for Vision-Language-Action Models
  2. 2025Experiences from Benchmarking Vision-Language-Action Models for Robotic Manipulation
  3. 2025Exploring the Limits of Vision-Language-Action Manipulations in Cross-task Generalization
  4. 2026ManipArena: Comprehensive Real-world Evaluation of Reasoning-Oriented Generalist Robot Manipulation
  5. 2024Benchmarking Vision, Language, & Action Models on Robotic Learning Tasks
  6. 2026Grounding Sim-to-Real Generalization in Robotic Manipulation: An Empirical Study with Vision-Language-Action Models
  7. 2025From Intention to Execution: Probing the Generalization Boundaries of Vision-Language-Action Models
  8. 2026Benchmarking Vision-Language-Action Models on SO-101: Failure and Recovery Analysis