# RoboChallenge: Large-scale Real-robot Evaluation of Embodied Policies

arXiv: https://arxiv.org/abs/2510.17950 · HTML: https://mlsanity.com/paper/2510.17950/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Oct 2025.

## Built on by (most cited first, 30 of 30)

- [A Pragmatic VLA Foundation Model](https://mlsanity.com/paper/2601.18692/) (2026)
- [World Action Models: The Next Frontier in Embodied AI](https://mlsanity.com/paper/2605.12090/) (2026)
- [RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies](https://mlsanity.com/paper/2607.04434/) (2026)
- [StarVLA-α: Reducing Complexity in Vision-Language-Action Systems](https://mlsanity.com/paper/2604.11757/) (2026)
- [Towards Generalizable Robotic Manipulation in Dynamic Environments](https://mlsanity.com/paper/2603.15620/) (2026)
- [Wall-OSS-0.5 Technical Report](https://mlsanity.com/paper/2605.30877/) (2026)
- [World Action Models: A Survey](https://mlsanity.com/paper/2606.20781/) (2026)
- [GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluation](https://mlsanity.com/paper/2607.02642/) (2026)
- [ProcVLM: Learning Procedure-Grounded Progress Rewards for Robotic Manipulation](https://mlsanity.com/paper/2605.08774/) (2026)
- [Vision-Language-Action in Robotics: A Survey of Datasets, Benchmarks, and Data Engines](https://mlsanity.com/paper/2604.23001/) (2026)
- [PACE: Phase-Aware Chunk Execution for Robot Policies with Action Chunking](https://mlsanity.com/paper/2606.00537/) (2026)
- [Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends](https://mlsanity.com/paper/2606.01164/) (2026)
- [IG-RFT: An Interaction-Guided RL Framework for VLA Models in Long-Horizon Robotic Manipulation](https://mlsanity.com/paper/2602.20715/) (2026)
- [ManipArena: Comprehensive Real-world Evaluation of Reasoning-Oriented Generalist Robot Manipulation](https://mlsanity.com/paper/2603.28545/) (2026)
- [LABSHIELD: A Multimodal Benchmark for Safety-Critical Reasoning and Planning in Scientific Laboratories](https://mlsanity.com/paper/2603.11987/) (2026)
- [PRM-as-a-Judge 1.5: A Toolkit for Robot Process Assessment](https://mlsanity.com/paper/2608.14284/) (2026)
- [VLA-REPLICA: A Low-Cost, Reproducible Benchmark for Real-World Evaluation of Vision-Language-Action Models](https://mlsanity.com/paper/2605.20774/) (2026)
- [PhAIL: A Real-Robot VLA Benchmark and Distributional Methodology](https://mlsanity.com/paper/2605.29710/) (2026)
- [You Don't Need To Stay in The Loop: An Agentic Robotics Loop for Robot-Policy Improvement](https://mlsanity.com/paper/2608.07555/) (2026)
- [X-DiffVLA: X-Embodied Diffusion Action Heads for Vision-Language-Action Models](https://mlsanity.com/paper/2605.25044/) (2026)
- [Intelligent Automation for Embodied Benchmark Construction: Pipelines, Embodiments, Simulators, and Trends](https://mlsanity.com/paper/2606.12207/) (2026)
- [ATOM-Bench: A Real-World Benchmark for Atomic Skills and Compositional Generalization in Manipulation Policies](https://mlsanity.com/paper/2606.16826/) (2026)
- [RoboFollow: Unveiling the Instruction Following Mirage in Embodied Agents](https://mlsanity.com/paper/2609.25636/) (2026)
- [Grounding Sim-to-Real Generalization in Robotic Manipulation: An Empirical Study with Vision-Language-Action Models](https://mlsanity.com/paper/2603.22876/) (2026)
- [PACT: Self-Evolving Physical Safety Alignment for Diffusion Policies in Embodied Manipulation](https://mlsanity.com/paper/2606.08414/) (2026)
- [Towards Trustworthy Embodied Intelligence: A Systems Framework and Graded Trustworthiness Levels](https://mlsanity.com/paper/2607.26121/) (2026)
- [From Generation to Simulation: How Far Are World Models from Being True Simulators?](https://mlsanity.com/paper/2608.23070/) (2026)
- [One Policy, Many Embodiments: Unified Camera-Centric Action Geometry Pre-training for Heterogeneous Embodied Manipulation](https://mlsanity.com/paper/2608.26058/) (2026)
- [RoboFolDeX: A Physical-World Benchmark for Long-Horizon Robotic Manipulation of Deformable Objects](https://mlsanity.com/paper/2609.10243/) (2026)
- [EWAM: Emergent Depth-Wise Specialization in a Unified Embodied Model -- From Semantic Understanding through Visual Foresight to Action](https://mlsanity.com/paper/2609.39973/) (2026)

## Abstract (arXiv)

Testing on real machines is indispensable for robotic control algorithms. In the context of learning-based algorithms, especially VLA models, demand for large-scale evaluation, i.e. testing a large number of models on a large number of tasks, is becoming increasingly urgent. However, doing this right is highly non-trivial, especially when scalability and reproducibility is taken into account. In this report, we describe our methodology for constructing RoboChallenge, an online evaluation system to test robotic control algorithms, and our survey of recent state-of-the-art VLA models using our initial benchmark Table30.