mlsanityData to 14 Oct 2026

TrustRoboReward: Preference-Ordered Isotonic Score Editing for Multi-Paradigm Robot Reward Models

arXiv 2608.08491 ↗First posted 9 Aug 2026cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward models are a bottleneck for reinforcement learning in embodied AI. Long-horizon robotic manipulation requires scalable vision feedback beyond handcrafted rewards or task-specific annotations. Existing open-source VLM reward judges like RoboReward adopt simple 1--5 trajectory progress scoring, lacking pairwise preferences for RLHF, DPO and Bradley-Terry frameworks, while failing to optimize video scene understanding. Augmenting RoboReward with pairwise comparison and video-QA supervision causes inconsistency between pairwise preferences and pointwise scores, introducing training noise and hurting downstream performance---an issue aggregation methods such as TrustJudge cannot resolve. To address this, we propose TrustRoboReward, a multi-paradigm reward modeling framework equipped with Preference-Ordered Isotonic Score Editing (POISE). We construct a unified four-paradigm dataset with trajectory progress scoring (Score-A), video-QA answer quality scoring (Score-B), and their pairwise counterparts (Pair-A, Pair-B). Pairwise labels align better with human judgment than pointwise scores, inspiring us to calibrate pointwise scores to avoid score-pair reversals against pairwise preferences. POISE rectifies pointwise scores and eliminates cross-paradigm reversal conflicts unresolved by TrustJudge. Theoretically, POISE reduces score-pair reversal conflicts from 20.15% to 0%, whereas TrustJudge retains 20.46% conflicts on the same corpus. Evaluated on our benchmark, Qwen3-VL-4B trained with POISE achieves an overall reward score of 77.96%, nearly matching GPT-5-mini (78.09%, gap 0.13%) and outperforming the strongest RoboReward-4B baseline by 10.13%. It also lifts test-time score-pair consistency to 71.90%, exceeding RoboReward-4B (57.26%) and GPT-5-mini (68.09%). Integrating TrustJudge aggregation during inference boosts the overall score to 78.57%, surpassing the GPT-5-mini teacher model.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 47 of its 52 references that are in the map. First posted on arXiv 9 Aug 2026.

Builds on

47 of 52 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Generative Reward Models2 Oct
  2. 2024HybridFlow: A Flexible and Efficient RLHF Framework28 Sep
  3. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  4. 2023Eureka: Human-Level Reward Design via Coding Large Language Models19 Oct
  5. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning19 Oct
  6. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback1 Sep
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  8. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 39

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  7. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  9. 2025Qwen3-VL Technical Reportcited by 2,793
  10. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  11. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  12. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
27 more
  1. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  2. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 1,002
  3. 2025OpenAI GPT-5 System Cardcited by 989
  4. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  5. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 372
  6. 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 355
  7. 2024Training Software Engineering Agents and Verifiers with SWE-Gymcited by 336
  8. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 313
  9. 2025Improving Video Generation with Human Feedbackcited by 284
  10. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 204
  11. 2025World Simulation with Video Foundation Models for Physical AIcited by 200
  12. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 156
  13. 2024VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generationcited by 156
  14. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 149
  15. 2024SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learningcited by 136
  16. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 134
  17. 2024Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluationcited by 125
  18. 2025Improving Vision-Language-Action Model with Online Reinforcement Learningcited by 116
  19. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Modelscited by 86
  20. 2023Vision-Language Models as a Source of Rewardscited by 66
  21. 2026RoboReward: General-Purpose Vision-Language Reward Models for Roboticscited by 53
  22. 2024CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and Evolutioncited by 42
  23. 2025PAI-Bench: A Comprehensive Benchmark For Physical AIcited by 34
  24. 2025Self-Improving Embodied Foundation Modelscited by 32
  25. 2025Robot-R1: Reinforcement Learning for Enhanced Embodied Reasoning in Roboticscited by 21
  26. 2025TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Themcited by 20
  27. 2023Reinforcement Learning with Foundation Priors: Let the Embodied Agent Efficiently Learn on Its Owncited by 19

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Pairwise Feedback: Listwise Vision-Language Supervision for Preference-Based Reward Learning
  2. 2025VARP: Reinforcement Learning from Vision-Language Model Feedback with Agent Regularized Preferences
  3. 2026TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics
  4. 2026Freeform Preference Learning for Robotic Manipulation
  5. 2026Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons
  6. 2026SARM2: Multi-Task Stage Aware Reward Modeling for Self Improving Robotic Manipulation
  7. 2026Large Reward Models: Generalizable Online Robot Reward Generation with Vision-Language Models
  8. 2026Reducing Oracle Feedback with Vision-Language Embeddings for Preference-Based RL