IndustrialVLA-Bench: A Traceable Multi-Axis Evaluation of Open Robot Policy Models
On the map: World models → Embodied and robot world models
also in 1 more
Abstract (arXiv)
Open robot policies increasingly follow two paradigms: vision-language-action models (VLAs) directly map observations and instructions to actions, whereas world-action models (WAMs) incorporate learned video or world dynamics into policy learning or action generation. Although both target the same manipulation tasks and represent alternative design choices, they are commonly reported under different evaluation protocols, leaving their capability, robustness, language sensitivity, and deployment-cost trade-offs unclear. We present IndustrialVLA-Bench, an evidence-aware evaluation of six released VLA and WAM systems under a unified reporting schema. It separately evaluates clean capability on LIBERO, non-language robustness on LIBERO-Plus, instruction sensitivity on LIBERO-Para, and observed execution cost. Reported task scores aggregate three complete evaluations with distinct random seeds under a fixed checkpoint and inference configuration. Across all six systems, clean LIBERO averages differ by only 1.58 points, whereas robustness and paraphrase summaries span 14.62 and 31.08 points. Restricting every comparison to the three protocol-faithful systems preserves the effect (1.36, 14.62 and 23.10 points), so the diagnostic separation reported here does not depend on the weaker evidence tiers. We additionally report observed inference latency, peak memory, runtime mode, and an evidence status for every system. Protocol-faithful, near-reproduction, and pending-verification entries remain visibly separated; only protocol-faithful entries support strict comparisons. Rather than claiming universal superiority of either paradigm, IndustrialVLA-Bench provides traceable evidence for comparing released robot policies on shared practical criteria. Code and evaluation records are available at https://github.com/xiaoqi-7/IndustrialVLA-Bench.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Embodied and robot world models, one of 11 directions in World models. Builds on 18 of its 28 references that are in the map. First posted on arXiv 22 Sep 2026.
Builds on
18 of 28 references.
Closest ancestors · 2
Same direction, newest first
- 2026Do World Action Models Generalize Better than VLAs? A Robustness Study23 Mar
- 2026Fast-WAM: Do World Action Models Need Test-time Future Imagination?17 Mar
Other references · 16
- 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
- 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
- 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
- 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulationcited by 670
- 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 643
- 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
- 2026Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planningcited by 274
- 2025LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorizationcited by 166
- 2026LeRobot: An Open-Source Library for End-to-End Robot Learningcited by 70
- 2026Xiaomi-Robotics-0: An Open-Sourced Vision-Language-Action Model with Real-Time Executioncited by 42
- 2025VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Modelscited by 28
- 2026LIBERO-Para: A Diagnostic Benchmark and Metrics for Paraphrase Robustness in VLA Modelscited by 8
4 more
- 2026vla-eval: A Unified Evaluation Harness for Vision-Language-Action Modelscited by 7
- 2026HazardArena: Evaluating Semantic Safety in Vision-Language-Action Modelscited by 6
- 2026LangGap: Diagnosing and Closing the Language Gap in Vision-Language-Action Modelscited by 4
- 2025An Open-Source Software Toolkit & Benchmark Suite for the Evaluation and Adaptation of Multimodal Action Modelscited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluation
- 2025WorldGym: World Model as An Environment for Policy Evaluation
- 2026Benchmarking Vision-Language-Action Models on SO-101: Failure and Recovery Analysis
- 2024OpenVLA: An Open-Source Vision-Language-Action Model
- 2025WorldEval: World Model as Real-World Robot Policies Evaluator
- 2026Do World Models Make Better Robots? A Survey of Evaluation Benchmarks for Predictive Embodied Intelligence
- 2026RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluation
- 2026Prioritized Rollouts for Efficient World Model-based Vision-Language-Action Policy Optimization