mlsanityData to 14 Oct 2026

AutoEval: Autonomous Evaluation of Generalist Robot Manipulation Policies in the Real World

arXiv 2503.24278 ↗First posted 31 Mar 2025cs.RO cs.AIcited by 48

Not on the map. Why?

Abstract (arXiv)

Scalable and reproducible policy evaluation has been a long-standing challenge in robot learning. Evaluations are critical to assess progress and build better policies, but evaluation in the real world, especially at a scale that would provide statistically reliable results, is costly in terms of human time and hard to obtain. Evaluation of increasingly generalist robot policies requires an increasingly diverse repertoire of evaluation environments, making the evaluation bottleneck even more pronounced. To make real-world evaluation of robotic policies more practical, we propose AutoEval, a system to autonomously evaluate generalist robot policies around the clock with minimal human intervention. Users interact with AutoEval by submitting evaluation jobs to the AutoEval queue, much like how software jobs are submitted with a cluster scheduling system, and AutoEval will schedule the policies for evaluation within a framework supplying automatic success detection and automatic scene resets. We show that AutoEval can nearly fully eliminate human involvement in the evaluation process, permitting around the clock evaluations, and the evaluation results correspond closely to ground truth evaluations conducted by hand. To facilitate the evaluation of generalist policies in the robotics community, we provide public access to multiple AutoEval scenes in the popular BridgeData robot setup with WidowX robot arms. In the future, we hope that AutoEval scenes can be set up across institutions to form a diverse and distributed evaluation network.

Key facts. Cited by 48 papers in the map (data through 14 Oct 2026). Builds on 66 of its 86 references that are in the map. First posted on arXiv 31 Mar 2025.

Builds on

66 of 86 references.

References · 66

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  2. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  3. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
  4. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  5. 2024Octo: An Open-Source Generalist Robot Policycited by 1,484
  6. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,410
  7. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  8. 2023PaLM 2 Technical Reportcited by 1,162
  9. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  10. 2018DeepMind Control Suitecited by 956
  11. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
  12. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 852
38 more
  1. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  2. 2024PaliGemma: A versatile 3B VLM for transfercited by 696
  3. 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulationcited by 691
  4. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 689
  5. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 665
  6. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 657
  7. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 595
  8. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 584
  9. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 495
  10. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 407
  11. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 353
  12. 2024Robotic Control via Embodied Chain-of-Thought Reasoningcited by 346
  13. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 327
  14. 2024Latent Action Pretraining from Videoscited by 321
  15. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Modelscited by 269
  16. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 250
  17. 2023NoMaD: Goal Masked Diffusion Policies for Navigation and Explorationcited by 248
  18. 2018Unity: A General Platform for Intelligent Agentscited by 236
  19. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 229
  20. 2024ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AIcited by 229
  21. 2023Habitat 3.0: A Co-Habitat for Humans, Avatars and Robotscited by 228
  22. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 209
  23. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 206
  24. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 197
  25. 2021iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Taskscited by 194
  26. 2020RoboTHOR: An Open Simulation-to-Real Embodied AI Platformcited by 193
  27. 2022GNM: A General Navigation Model to Drive Any Robotcited by 171
  28. 2024BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulationcited by 148
  29. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 138
  30. 2023FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulationcited by 133
  31. 2020SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulationcited by 121
  32. 2024Scaling Cross-Embodied Learning: One Policy for Manipulation, Navigation, Locomotion and Aviationcited by 114
  33. 2020CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learningcited by 109
  34. 2023HomeRobot: Open-Vocabulary Mobile Manipulationcited by 106
  35. 2015Benchmarking in Manipulation Research: The YCB Object and Model Set and Benchmarking Protocolscited by 99
  36. 20226-DoF Pose Estimation of Household Objects for Robotic Manipulation: An Accessible Dataset and Benchmarkcited by 86
  37. 2021FlingBot: The Unreasonable Effectiveness of Dynamic Manipulation for Cloth Unfoldingcited by 81
  38. 2024FMB: a Functional Manipulation Benchmark for Generalizable Robotic Learningcited by 80

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 19; 2026: 29.

48 papers.

Sort:

By area:

  1. 2025Genie Envisioner: A Unified World Foundation Platform for Robotic Manipulationcited by 137
  2. 2026RoboCasa365: A Large-Scale Simulation Framework for Training and Benchmarking Generalist Robotscited by 79
  3. 2025CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsificationcited by 74
  4. 2025WorldEval: World Model as Real-World Robot Policies Evaluatorcited by 68
  5. 2025RoboArena: Distributed Real-World Evaluation of Generalist Robot Policiescited by 64
  6. 2025Embodied AI Agents: Modeling the Worldcited by 61
  7. 2026Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisonscited by 60
  8. 2025PolaRiS: Scalable Real-to-Sim Evaluations for Generalist Robot Policiescited by 33
  9. 2026Interactive World Simulator for Robot Policy Training and Evaluationcited by 31
  10. 2025Real-to-Sim Robot Policy Evaluation with Gaussian Splatting Simulation of Soft-Body Interactionscited by 30
  11. 2026World-Gymnast: Training Robots with Reinforcement Learning in a World Modelcited by 27
  12. 2025Humanoid Everyday: A Comprehensive Robotic Dataset for Open-World Humanoid Manipulationcited by 25
36 more
  1. 2025RobotArena ∞: Scalable Robot Benchmarking via Real-to-Sim Translationcited by 18
  2. 2025Reliable and Scalable Robot Policy Evaluation with Imperfect Simulatorscited by 14
  3. 2025Real-is-Sim: Bridging the Sim-to-Real Gap with a Dynamic Digital Twincited by 12
  4. 2025I-FailSense: Towards General Robotic Failure Detection with Vision-Language Modelscited by 12
  5. 2026AutoResearch AI: Towards AI-Powered Research Automation for Scientific Discoverycited by 10
  6. 2025CORE: Full-Path Evaluation of LLM Agents Beyond Final Statecited by 9
  7. 2026World2Act: Latent Action Post-Training from World Model Dynamicscited by 8
  8. 2025Survey of Vision-Language-Action Models for Embodied Manipulationcited by 7
  9. 2025Towards High-Consistency Embodied World Model with Multi-View Trajectory Videoscited by 6
  10. 2026RADAR: Benchmarking Vision-Language-Action Generalization via Real-World Dynamics, Spatial-Physical Intelligence, and Autonomous Evaluationcited by 4
  11. 2026RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluationcited by 4
  12. 2025"Are We Done Yet?": A Vision-Based Judge for Autonomous Task Completion of Computer Use Agentscited by 3
  13. 2026PRM-as-a-Judge 1.5: A Toolkit for Robot Process Assessmentcited by 3
  14. 2025Imagine, Verify, Execute: Memory-guided Agentic Exploration with Vision-Language Modelscited by 2
  15. 2025AURA: Autonomous Upskilling with Retrieval-Augmented Agentscited by 2
  16. 2025CRAFT: Coaching Reinforcement Learning Autonomously using Foundation Models for Multi-Robot Coordination Taskscited by 2
  17. 2026Benchmarking Affordance Generalization with BusyBoxcited by 2
  18. 2026VLA-REPLICA: A Low-Cost, Reproducible Benchmark for Real-World Evaluation of Vision-Language-Action Modelscited by 2
  19. 2026iMaC: Translating Actions into Motion and Contact Images for Embodied World Modelscited by 2
  20. 2026World Models as an Intermediary between Agents and the Real Worldcited by 1
  21. 2026ATOM-Bench: A Real-World Benchmark for Atomic Skills and Compositional Generalization in Manipulation Policiescited by 1
  22. 2026ArmnetBench v0.1: Parallel Real-World Evaluation of Manipulation Policies on a Low-Cost Arm Farmcited by 1
  23. 2026The Embodiment Gap in Robot Foundation Modelscited by 1
  24. 2026AdaHVLA: Adaptive Harnesses for Long-Horizon Vision-Language-Action Executioncited by 1
  25. 2026Self-Evolving Coding Agents: From Digital Programs to Physical-World Intelligencecited by 1
  26. 2026PyBatchRender: A Python Library for Batched 3D Rendering at Up to One Million FPScited by 0
  27. 2026A Practical Recipe Towards Improving Sim-and-Real Correlation for VLA Evaluationcited by 0
  28. 2026Critical Interval MSE: Toward Reliable Offline Validation for Robot Manipulation Policiescited by 0
  29. 2026Does Imitation Learning Preserve Temporal Robustness in Dexterous Manipulation? An Expert-Learner Comparison Across Task Execution Speedscited by 0
  30. 2026R2S-Eval: Robot Evaluation with Real-to-Sim Calibration via Vision-Language Modelscited by 0
  31. 2026FailBench: How Reliable are VLMs at Judging Robot Task Success?cited by 0
  32. 2026No Free Checker: A Survey of Verifiers for Robot Policiescited by 0
  33. 2026From Rollout to Reset: A Graph-Based Harness for Autonomous Long-Horizon Manipulation Evaluationcited by 0
  34. 2026Anatomy of a Closed-Loop Collapse: A Causal Case Study of a Compressed VLA Policycited by 0
  35. 2026ReVeal: A Reconstruction-Aware Real-to-Sim Framework for VLA Policy Evaluationcited by 0
  36. 2026Hierarchical Residual Policy Learning for Real-World Mobile Manipulation with Sparse Human Guidancecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Active Real-World Factor-Based Evaluation for Generalist Robot Policies
  2. 2026RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies
  3. 2026Eval-Actions: Fine-Grained Execution Quality Evaluation for Robotic Manipulation
  4. 2025Score the Steps, Not Just the Goal: VLM-Based Subgoal Evaluation for Robotic Manipulation
  5. 2026Beyond Binary Success: Sample-Efficient and Statistically Rigorous Robot Policy Comparison
  6. 2026GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluation
  7. 2026dWorldEval: Scalable Robotic Policy Evaluation via Discrete Diffusion World Model
  8. 2026ManipArena: Comprehensive Real-world Evaluation of Reasoning-Oriented Generalist Robot Manipulation