mlsanityData to 7 Oct 2026

Same Trajectory, Contradictory Rewards (ROBORMBENCH): Paraphrase Fragility in Vision Language Reward Models

arXiv 2609.05401 ↗First posted 4 Sep 2026cs.RO cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Vision-language models are increasingly used as reward functions for robotic learning, but this role requires paraphrase invariance: the same trajectory should receive the same reward under semantically equivalent goal descriptions. We show that current VLM reward models often violate this property. Paraphrasing the instruction alone can substantially change predicted progress scores, and can even flip identical robot behavior between failure and success. To measure this failure mode, we introduce ROBORMBENCH, a benchmark with 2,390 real-robot trajectories, ground-truth progress labels, and 21,673 verified paraphrases spanning lexical, syntactic, and action-goal rewrites. Across proprietary and open-source VLMs, paraphrase-induced instability is widespread and severe, grows under more divergent rewrites, and is not reliably reduced by scale or explicit reasoning. Dedicated reward models trained with trajectory-grounded supervision are substantially more stable. These results show that paraphrase robustness is a core requirement for reliable VLM-based reward modeling in robotics.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 57 references that are in the map. First posted on arXiv 4 Sep 2026.

Builds on

43 of 57 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025VARP: Reinforcement Learning from Vision-Language Model Feedback with Agent Regularized Preferences18 Mar
  2. 2024Real-World Offline Reinforcement Learning from Vision Language Model Feedback8 Nov
  3. 2024Vision Language Models are In-Context Value Learners7 Nov
  4. 2024Code as Reward: Empowering Reinforcement Learning with VLMs7 Feb
  5. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback6 Feb
  6. 2023Vision-Language Models as a Source of Rewards14 Dec
  7. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning19 Oct

Other references · 36

  1. 2017Deep reinforcement learning from human preferencescited by 3,846
  2. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  3. 2023Let's Verify Step by Stepcited by 3,598
  4. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  5. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  6. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  7. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  8. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  9. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 312
  10. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  11. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 168
  12. 2024Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learningcited by 153
24 more
  1. 2025ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policycited by 139
  2. 2023Vision-Language Models as Success Detectorscited by 138
  3. 2016EPOpt: Learning Robust Neural Network Policies Using Model Ensemblescited by 136
  4. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  5. 2024SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learningcited by 113
  6. 2020The Ingredients of Real-World Robotic Reinforcement Learningcited by 111
  7. 2024GRAPE: Generalizing Robot Policy via Preference Alignmentcited by 106
  8. 2024Steering Your Generalists: Improving Robotic Foundation Models via Value Guidancecited by 100
  9. 2025A Survey on Vision-Language-Action Models: An Action Tokenization Perspectivecited by 96
  10. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Modelscited by 85
  11. 2024Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbonecited by 71
  12. 2021Legged Robots that Keep on Learning: Fine-Tuning Locomotion Policies in the Real Worldcited by 64
  13. 2025RoboArena: Distributed Real-World Evaluation of Generalist Robot Policiescited by 63
  14. 2024FLaRe: Achieving Masterful and Adaptive Robot Policies with Large-Scale Reinforcement Learning Fine-Tuningcited by 54
  15. 2025ReWiND: Language-Guided Rewards Teach Robot Policies without New Demonstrationscited by 52
  16. 2026Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisonscited by 51
  17. 2024VLM-RL: A Unified Vision Language Models and Reinforcement Learning Framework for Safe Autonomous Drivingcited by 49
  18. 2025Robo-Dopamine: General Process Reward Modeling for High-Precision Robotic Manipulationcited by 37
  19. 2024Rank2Reward: Learning Shaped Reward Functions from Passive Videocited by 30
  20. 2024Continuously Improving Mobile Manipulation with Autonomous Real-World RLcited by 20
  21. 2025VLRMBench: A Comprehensive and Challenging Benchmark for Vision-Language Reward Modelscited by 19
  22. 2025When Language Overrules: Revealing Text Dominance in Multimodal Large Language Modelscited by 16
  23. 2024Adapt2Reward: Adapting Video-Language Models to Generalizable Robotic Rewards via Failure Promptscited by 9
  24. 2024Video2Reward: Generating Reward Function from Videos for Legged Robot Behavior Learningcited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026LIBERO-Para: A Diagnostic Benchmark and Metrics for Paraphrase Robustness in VLA Models
  2. 2026RoboReward: General-Purpose Vision-Language Reward Models for Robotics
  3. 2025Training-free Generation of Temporally Consistent Rewards from VLMs
  4. 2024Video-Language Critic: Transferable Reward Functions for Language-Conditioned Robotics
  5. 2025A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learning
  6. 2026MARVL: Multi-Stage Guidance for Robotic Manipulation via Vision-Language Models
  7. 2026RoboSemanticBench: Diagnosing Semantic Grounding in Action Prediction for VLA Models
  8. 2025ViLBench: A Suite for Vision-Language Process Reward Modeling