# RoboReward: General-Purpose Vision-Language Reward Models for Robotics

arXiv: https://arxiv.org/abs/2601.00675 · HTML: https://mlsanity.com/paper/2601.00675/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jan 2026.

## Built on by (most cited first, 30 of 53)

- [Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons](https://mlsanity.com/paper/2603.02115/) (2026)
- [TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics](https://mlsanity.com/paper/2602.19313/) (2026)
- [Towards Long-Lived Robots: Continual Learning VLA Models via Reinforcement Fine-Tuning](https://mlsanity.com/paper/2602.10503/) (2026)
- [Large Reward Models: Generalizable Online Robot Reward Generation with Vision-Language Models](https://mlsanity.com/paper/2603.16065/) (2026)
- [Learning Actionable Manipulation Recovery via Counterfactual Failure Synthesis](https://mlsanity.com/paper/2603.13528/) (2026)
- [PRM-as-a-Judge: A Dense Evaluation Paradigm for Fine-Grained Robotic Auditing](https://mlsanity.com/paper/2603.21669/) (2026)
- [ProcVLM: Learning Procedure-Grounded Progress Rewards for Robotic Manipulation](https://mlsanity.com/paper/2605.08774/) (2026)
- [RynnValue: Scaling Robotic Value Foundation Models with Temporal Distance](https://mlsanity.com/paper/2608.09853/) (2026)
- [ACE-Brain-0.5: A Unified Embodied Foundational Model for Physical Agentic AI](https://mlsanity.com/paper/2607.04426/) (2026)
- [Critic in the Loop: A Tri-System VLA Framework for Robust Long-Horizon Manipulation](https://mlsanity.com/paper/2603.05185/) (2026)
- [DriveVLM-RL: Neuroscience-Inspired Reinforcement Learning with Vision-Language Models for Safe and Deployable Autonomous Driving](https://mlsanity.com/paper/2603.18315/) (2026)
- [SAIL: Test-Time Scaling for In-Context Imitation Learning with VLM](https://mlsanity.com/paper/2603.08269/) (2026)
- [SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learning](https://mlsanity.com/paper/2603.28730/) (2026)
- [RDA: Reward Design Agent for Reinforcement Learning](https://mlsanity.com/paper/2606.01672/) (2026)
- [VeriSpace: Spatially Grounded Action Verification for Vision-Language-Action Models](https://mlsanity.com/paper/2606.10568/) (2026)
- [DriveJudge: Rethinking Autonomous Driving Evaluation with Vision-Language Models](https://mlsanity.com/paper/2606.17362/) (2026)
- [RARM: Confidence-Gated Progress Reward Modeling for RL in Manipulation](https://mlsanity.com/paper/2606.22027/) (2026)
- [ELASTIC: Efficiently Learning to Adaptively Scale Test-Time Compute for Generative Control Policies](https://mlsanity.com/paper/2606.31132/) (2026)
- [PRM-as-a-Judge 1.5: A Toolkit for Robot Process Assessment](https://mlsanity.com/paper/2608.14284/) (2026)
- [Dual-Granularity Contrastive Reward via Generated Episodic Guidance for Efficient Embodied RL](https://mlsanity.com/paper/2602.12636/) (2026)
- [MARVL: Multi-Stage Guidance for Robotic Manipulation via Vision-Language Models](https://mlsanity.com/paper/2602.15872/) (2026)
- [Rewarding DINO: Predicting Dense Rewards with Vision Foundation Models](https://mlsanity.com/paper/2603.16978/) (2026)
- [SARM2: Multi-Task Stage Aware Reward Modeling for Self Improving Robotic Manipulation](https://mlsanity.com/paper/2606.10305/) (2026)
- [Latent Goal Prediction from Language for Model-Based Planning](https://mlsanity.com/paper/2606.20627/) (2026)
- [DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation](https://mlsanity.com/paper/2607.13033/) (2026)
- [ContactGuard: Pre-Contact Execution Monitoring with Action-Conditioned Latent World Models](https://mlsanity.com/paper/2608.13438/) (2026)
- [Prism-GRPO: Faster VLA Policy Optimization via Splitting Same-outcome Groups](https://mlsanity.com/paper/2608.17423/) (2026)
- [World Models as an Intermediary between Agents and the Real World](https://mlsanity.com/paper/2602.00785/) (2026)
- [Beyond Pixels: Learning Invariant Rewards for Real-World Robotics From a Few Demonstrations](https://mlsanity.com/paper/2605.22123/) (2026)
- [From Demonstrations to Rewards: Test-Time Prompt Optimization for VLM Reward Models](https://mlsanity.com/paper/2606.00083/) (2026)

## Abstract (arXiv)

A well-designed reward is critical for effective reinforcement learning-based policy improvement. In real-world robotics, obtaining such rewards typically requires either labor-intensive human labeling or brittle, handcrafted objectives. Vision-language models (VLMs) have shown promise as automatic reward models, yet their effectiveness on real robot tasks is poorly understood. In this work, we aim to close this gap by introducing (1) RoboReward, a robotics reward dataset and benchmark built on large-scale real-robot corpora from Open X-Embodiment (OXE) and RoboArena, and (2) vision-language reward models trained on this dataset (RoboReward 4B/8B). Because OXE is success-heavy and lacks failure examples, we propose a negative examples data augmentation pipeline that generates calibrated negative and near-misses via counterfactual relabeling of successful episodes and temporal clipping to create partial-progress outcomes from the same videos. Using this framework, we build a large training and evaluation dataset spanning diverse tasks and embodiments to test whether state-of-the-art VLMs can reliably provide rewards for robot learning. Our evaluation of open and proprietary VLMs finds that no model excels across tasks, highlighting substantial room for improvement. We then train general-purpose 4B- and 8B-parameter models that outperform much larger VLMs in assigning rewards for short-horizon robotic tasks. Finally, we deploy the 8B model in real-robot reinforcement learning and find that it improves policy learning over Gemini Robotics-ER 1.5 while narrowing the gap to RL training with human-provided rewards. We release the full dataset, trained reward models, and evaluation suite on our website to advance the development of general-purpose reward models in robotics: https://crfm.stanford.edu/helm/robo-reward-bench (project website).