mlsanityData to 14 Oct 2026

RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies

arXiv 2506.18123 ↗First posted 22 Jun 2025cs.RO cs.LGcited by 64

Not on the map. Why?

Cited by 25 in Vision-language-action models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Comprehensive, unbiased, and comparable evaluation of modern generalist policies is uniquely challenging: existing approaches for robot benchmarking typically rely on heavy standardization, either by specifying fixed evaluation tasks and environments, or by hosting centralized ''robot challenges'', and do not readily scale to evaluating generalist policies across a broad range of tasks and environments. In this work, we propose RoboArena, a new approach for scalable evaluation of generalist robot policies in the real world. Instead of standardizing evaluations around fixed tasks, environments, or locations, we propose to crowd-source evaluations across a distributed network of evaluators. Importantly, evaluators can freely choose the tasks and environments they evaluate on, enabling easy scaling of diversity, but they are required to perform double-blind evaluations over pairs of policies. Then, by aggregating preference feedback from pairwise comparisons across diverse tasks and environments, we can derive a ranking of policies. We instantiate our approach across a network of evaluators at seven academic institutions using the DROID robot platform. Through more than 600 pairwise real-robot evaluation episodes across seven generalist policies, we demonstrate that our crowd-sourced approach can more accurately rank the performance of existing generalist policies than conventional, centralized evaluation approaches, while being more scalable, resilient, and trustworthy. We open our evaluation network to the community and hope that it can enable more accessible comparisons of generalist robot policies.

Key facts. Cited by 64 papers in the map (data through 14 Oct 2026). Builds on 54 of its 72 references that are in the map. First posted on arXiv 22 Jun 2025.

Builds on

54 of 72 references.

References · 54

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  2. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  3. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  4. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
  5. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  6. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
  7. 2024Octo: An Open-Source Generalist Robot Policycited by 1,484
  8. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,410
  9. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,225
  10. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  11. 2018DeepMind Control Suitecited by 956
  12. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
38 more
  1. 2024PaliGemma: A versatile 3B VLM for transfercited by 696
  2. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 689
  3. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 665
  4. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 657
  5. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 584
  6. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 495
  7. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 440
  8. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 407
  9. 2025AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systemscited by 405
  10. 2024Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robotscited by 394
  11. 2024Robotic Control via Embodied Chain-of-Thought Reasoningcited by 346
  12. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 339
  13. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 336
  14. 20243D-VLA: A 3D Vision-Language-Action Generative World Modelcited by 328
  15. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 270
  16. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 250
  17. 2025DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Controlcited by 195
  18. 2023RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shotcited by 183
  19. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 169
  20. 2021BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environmentscited by 165
  21. 2024Behavior Generation with Latent Actionscited by 160
  22. 2023FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulationcited by 133
  23. 2024THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulationcited by 133
  24. 2024Scaling Cross-Embodied Learning: One Policy for Manipulation, Navigation, Locomotion and Aviationcited by 114
  25. 2023On Bringing Robots Homecited by 101
  26. 2024GenAI Arena: An Open Evaluation Platform for Generative Modelscited by 82
  27. 2024FMB: a Functional Manipulation Benchmark for Generalizable Robotic Learningcited by 80
  28. 2018Robot Learning in Homes: Improving Generalization and Reducing Dataset Biascited by 75
  29. 2019IKEA Furniture Assembly Environment for Long-Horizon Complex Manipulation Taskscited by 69
  30. 2023DexArt: Benchmarking Generalizable Dexterous Manipulation with Articulated Objectscited by 66
  31. 2025AutoEval: Autonomous Evaluation of Generalist Robot Manipulation Policies in the Real Worldcited by 48
  32. 2024Robot Utility Models: General Policies for Zero-Shot Deployment in New Environmentscited by 44
  33. 2024From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessonscited by 42
  34. 2016Analysis and Observations from the First Amazon Picking Challengecited by 41
  35. 2016The Robotarium: A remotely accessible swarm robotics research testbedcited by 24
  36. 2022RB2: Robotic Manipulation Benchmarking with a Twistcited by 23
  37. 2024Robot Learning as an Empirical Science: Best Practices for Policy Evaluationcited by 23
  38. 2021Real Robot Challenge: A Robotics Competition in the Cloudcited by 14

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 15; 2026: 49.

64 papers.

Sort:

By area:

  1. 2025Ctrl-World: A Controllable Generative World Model for Robot Manipulationcited by 167
  2. 2026A Pragmatic VLA Foundation Modelcited by 103
  3. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 84
  4. 2026RoboCasa365: A Large-Scale Simulation Framework for Training and Benchmarking Generalist Robotscited by 79
  5. 2025Embodied AI Agents: Modeling the Worldcited by 61
  6. 2026Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisonscited by 60
  7. 2026World Action Models: The Next Frontier in Embodied AIcited by 50
  8. 2026Hierarchical Planning with Latent World Modelscited by 43
  9. 2026RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policiescited by 40
  10. 2025PolaRiS: Scalable Real-to-Sim Evaluations for Generalist Robot Policiescited by 33
  11. 2026Interactive World Simulator for Robot Policy Training and Evaluationcited by 31
  12. 2025Real-to-Sim Robot Policy Evaluation with Gaussian Splatting Simulation of Soft-Body Interactionscited by 30
38 more
  1. 2026RDT2: Exploring the Scaling Limit of UMI Data Towards Zero-Shot Cross-Embodiment Generalizationcited by 30
  2. 2026RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policiescited by 24
  3. 2025Robust Finetuning of Vision-Language-Action Robot Policies via Parameter Mergingcited by 22
  4. 2025REALM: A Real-to-Sim Validated Benchmark for Generalization in Robotic Manipulationcited by 15
  5. 2026SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulationcited by 15
  6. 2025Reliable and Scalable Robot Policy Evaluation with Imperfect Simulatorscited by 14
  7. 2025Improving Pre-Trained Vision-Language-Action Policies with Model-Based Searchcited by 13
  8. 2025PEEK: Guiding and Minimal Image Representations for Zero-Shot Generalization of Robot Manipulation Policiescited by 12
  9. 2026World Action Models: A Surveycited by 12
  10. 2025CORE: Full-Path Evaluation of LLM Agents Beyond Final Statecited by 9
  11. 2025Survey of Vision-Language-Action Models for Embodied Manipulationcited by 7
  12. 2026SC3-Eval: Evaluating Robot Foundation Models via Self-Consistent Video Generationcited by 7
  13. 2025Embodied AI: Emerging Risks and Opportunities for Policy Actioncited by 5
  14. 2026ManipArena: Comprehensive Real-world Evaluation of Reasoning-Oriented Generalist Robot Manipulationcited by 5
  15. 2026A Definition and Roadmap for World Modelscited by 5
  16. 2026Colosseum V2: Benchmarking Generalization for Vision-Language-Action Modelscited by 4
  17. 2026RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluationcited by 4
  18. 2025RoboManipBaselines: A Unified Framework for Imitation Learning in Robotic Manipulation across Real and Simulation Environmentscited by 3
  19. 2025SPEAR-1: Scaling Beyond Robot Demonstrations via 3D Understandingcited by 2
  20. 2026Grounding Robot Generalization in Training Data via Retrieval-Augmented VLMscited by 2
  21. 2026VLA-REPLICA: A Low-Cost, Reproducible Benchmark for Real-World Evaluation of Vision-Language-Action Modelscited by 2
  22. 2026PhAIL: A Real-Robot VLA Benchmark and Distributional Methodologycited by 2
  23. 2026VISTA: Vision-Grounded and Physics-Validated Adaptation of UMI data for VLA Trainingcited by 2
  24. 2026iMaC: Translating Actions into Motion and Contact Images for Embodied World Modelscited by 2
  25. 2026DuoBench: A Reproducible Benchmark for Bimanual Manipulation in Simulation and the Real Worldcited by 2
  26. 2026RoboPlayground: Democratizing Robotic Evaluation through Structured Physical Domainscited by 1
  27. 2026From Where Things Are to What They Are For: Benchmarking Spatial-Functional Intelligence in Multimodal LLMscited by 1
  28. 2026FATE-VLA:Failue-aware test generation for vision-language-action modelscited by 1
  29. 2026ATOM-Bench: A Real-World Benchmark for Atomic Skills and Compositional Generalization in Manipulation Policiescited by 1
  30. 2026FurnitureVLA: Learning Long-Horizon Bimanual Furniture Assembly with Vision-Language-Action Modelcited by 1
  31. 2026DREAMSTEER: Latent World Models Can Steer VLA Policies During Deployment Without Any Finetuningcited by 1
  32. 2026ArmnetBench v0.1: Parallel Real-World Evaluation of Manipulation Policies on a Low-Cost Arm Farmcited by 1
  33. 2026Regression Test Selection for Updated Capability Modules in Compositional ML Systems via Atomic-Quality Probescited by 0
  34. 2026A Practical Recipe Towards Improving Sim-and-Real Correlation for VLA Evaluationcited by 0
  35. 2026Assistron: Bayesian Shared Autonomy with Off-the-shelf Vision-Language-Action Modelscited by 0
  36. 2026SPACE: Enabling Learning from Cross-Robot Data Toward Generalist Policiescited by 0
  37. 2026Critical Interval MSE: Toward Reliable Offline Validation for Robot Manipulation Policiescited by 0
  38. 2026Robustness of Robotic Manipulation: Foundations and Frontierscited by 0

50 most cited of 64 All 64 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Active Real-World Factor-Based Evaluation for Generalist Robot Policies
  2. 2026Beyond Binary Success: Sample-Efficient and Statistically Rigorous Robot Policy Comparison
  3. 2026dWorldEval: Scalable Robotic Policy Evaluation via Discrete Diffusion World Model
  4. 2026GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluation
  5. 2025RoboChallenge: Large-scale Real-robot Evaluation of Embodied Policies
  6. 2025WorldEval: World Model as Real-World Robot Policies Evaluator
  7. 2025RobotArena ∞: Scalable Robot Benchmarking via Real-to-Sim Translation
  8. 2026X2Real: an eXtensive simulation benchmark for real-world generalist policies ↗