# PRM-as-a-Judge 1.5: A Toolkit for Robot Process Assessment

arXiv: https://arxiv.org/abs/2608.14284 · HTML: https://mlsanity.com/paper/2608.14284/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 54 of its 63 references that are in the map.
- First posted on arXiv 14 Aug 2026.

## Builds on (most cited first, 30 of 54)

- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks](https://mlsanity.com/paper/2112.03227/) (2021)
- [On Evaluation of Embodied Navigation Agents](https://mlsanity.com/paper/1807.06757/) (2018)
- [RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation](https://mlsanity.com/paper/2506.18088/) (2025)
- [robosuite: A Modular Simulation Framework and Benchmark for Robot Learning](https://mlsanity.com/paper/2009.12293/) (2020)
- [Evaluating Real-World Robot Manipulation Policies in Simulation](https://mlsanity.com/paper/2405.05941/) (2024)
- [Time-Contrastive Networks: Self-Supervised Learning from Video](https://mlsanity.com/paper/1704.06888/) (2017)
- [RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots](https://mlsanity.com/paper/2406.02523/) (2024)
- [VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training](https://mlsanity.com/paper/2210.00030/) (2022)
- [Fast-WAM: Do World Action Models Need Test-time Future Imagination?](https://mlsanity.com/paper/2603.16666/) (2026)
- [ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills](https://mlsanity.com/paper/2302.04659/) (2023)
- [X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Model](https://mlsanity.com/paper/2510.10274/) (2025)
- [LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models](https://mlsanity.com/paper/2510.13626/) (2025)
- [LIV: Language-Image Representations and Rewards for Robotic Control](https://mlsanity.com/paper/2306.00958/) (2023)
- [VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Tasks](https://mlsanity.com/paper/2412.18194/) (2024)
- [VLA-Adapter: An Effective Paradigm for Tiny-Scale Vision-Language-Action Model](https://mlsanity.com/paper/2509.09372/) (2025)
- [GigaWorld-Policy: An Efficient Action-Centered World--Action Model](https://mlsanity.com/paper/2603.17240/) (2026)
- [RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concrete](https://mlsanity.com/paper/2502.21257/) (2025)
- [Spatial Forcing: Implicit Spatial Representation Alignment for Vision-language-action Model](https://mlsanity.com/paper/2510.12276/) (2025)
- [Vision Language Models are In-Context Value Learners](https://mlsanity.com/paper/2411.04549/) (2024)
- [A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learning](https://mlsanity.com/paper/2509.15937/) (2025)
- [Unified 4D World Action Modeling from Video Priors with Asynchronous Denoising](https://mlsanity.com/paper/2604.26694/) (2026)
- [VLA-RFT: Vision-Language-Action Reinforcement Fine-tuning with Verified Rewards in World Simulators](https://mlsanity.com/paper/2510.00406/) (2025)
- [Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons](https://mlsanity.com/paper/2603.02115/) (2026)
- [RoboReward: General-Purpose Vision-Language Reward Models for Robotics](https://mlsanity.com/paper/2601.00675/) (2026)
- [LDA-1B: Scaling Latent Dynamics Action Model via Universal Embodied Data Ingestion](https://mlsanity.com/paper/2602.12215/) (2026)
- [InternVLA-A1: Unifying Understanding, Generation and Action for Robotic Manipulation](https://mlsanity.com/paper/2601.02456/) (2026)
- [AutoEval: Autonomous Evaluation of Generalist Robot Manipulation Policies in the Real World](https://mlsanity.com/paper/2503.24278/) (2025)
- [AHA-WAM:Asynchronous Horizon-Adaptive World-Action Modeling with Observation-Guided Context Routing](https://mlsanity.com/paper/2606.09811/) (2026)

## Built on by (most cited first, 3 of 3)

- [SARFusion: Scene-Aware Routing Fusion for Robust Camera-LiDAR 3D Object Detection](https://mlsanity.com/paper/2609.29235/) (2026)
- [RoboCoach: World Models as Active Coaches for Compositional Robot Skills](https://mlsanity.com/paper/2609.39685/) (2026)
- [Genesis: Visual Augmentation Primitives for Multimodal Chain-of-Thought Reasoning](https://mlsanity.com/paper/2610.15565/) (2026)

## Abstract (arXiv)

Fine-grained robotic evaluation matters for understanding embodied models, going beyond binary success rates and rule-based process scores. We present PRM-as-a-Judge 1.5, a toolkit for robot process assessment that turns rollout videos into dense progress curves and derives multiple fine metrics. PRM-as-a-Judge 1.5 introduces three metrics, building on version 1.0, that characterize failure-side progress, post-drawdown recovery, and success-side execution quality, helping users understand embodied model capability. Based on the rollout videos from benchmarks, we perform a comprehensive assessment of the embodied models, providing some fine-grained metric results and key findings. We further introduce RoboPulse++ to evaluate the reliability of process reward models (PRM), providing evaluators with a more accurate testing platform. Moreover, we release a user-friendly assessment suite, including the benchmark, metric implementation, and visualization tools, to support reproducible manipulation process evaluation. We call on the community to rethink how robots are evaluated and establish transparent, procedural, and reproducible assessment as a foundation for the next generation of embodied intelligence.