mlsanityData to 7 Oct 2026

Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons

arXiv 2603.02115 ↗First posted 2 Mar 2026cs.RO cs.AI cs.LGcited by 51

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

General-purpose robot reward models are typically trained to predict absolute task progress from expert demonstrations, providing only local, frame-level supervision. While effective for expert demonstrations, this paradigm scales poorly to large-scale robotics datasets where failed and suboptimal trajectories are abundant and assigning dense progress labels is ambiguous. We introduce Robometer, a scalable reward modeling framework that combines intra-trajectory progress supervision with inter-trajectory preference supervision. Robometer is trained with a dual objective: a frame-level progress loss that anchors reward magnitude on expert data, and a trajectory-comparison preference loss that imposes global ordering constraints across trajectories of the same task, enabling effective learning from both real and augmented failed trajectories. To support this formulation at scale, we curate RBM-1M, a reward-learning dataset comprising over one million trajectories spanning diverse robot embodiments and tasks, including substantial suboptimal and failure data. Across benchmarks and real-world evaluations, Robometer learns more generalizable reward functions than prior methods and improves robot learning performance across a diverse set of downstream applications. Code, model weights, and videos at https://robometer.github.io/.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 125 of its 145 references that are in the map. First posted on arXiv 2 Mar 2026.

Builds on

125 of 145 references.

Closest ancestors · 9

Same direction, newest first

  1. 2026TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics22 Feb
  2. 2025Masked IRL: LLM-Guided Reward Disambiguation from Demonstrations and Language18 Nov
  3. 2023Eureka: Human-Level Reward Design via Coding Large Language Models19 Oct
  4. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning19 Oct
  5. 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learning20 Sep
  6. 2021Learning Multimodal Rewards from Rankings27 Sep
  7. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  8. 2020Active Preference-Based Gaussian Process Regression for Reward Learning6 May
  9. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 116

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  4. 2025Qwen2.5-VL Technical Reportcited by 5,418
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  6. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  7. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  8. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  9. 2025Qwen3-VL Technical Reportcited by 2,534
  10. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  11. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  12. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
38 more
  1. 2016Generative Adversarial Imitation Learningcited by 1,576
  2. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  3. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  4. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  5. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 801
  6. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  7. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  8. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  9. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 643
  10. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  11. 2020Rescaling Egocentric Visioncited by 529
  12. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  13. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  14. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  15. 2026World Action Models are Zero-shot Policiescited by 411
  16. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  17. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  18. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 377
  19. 2019Learning Latent Plans from Playcited by 310
  20. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  21. 2025π*0.6: a VLA That Learns From Experiencecited by 280
  22. 2023Reward Design with Language Modelscited by 257
  23. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  24. 2024Reinforcement learningcited by 236
  25. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  26. 2024ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AIcited by 216
  27. 2025EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Videocited by 191
  28. 2025MolmoAct: Action Reasoning Models that can Reason in Spacecited by 189
  29. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  30. 2023RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shotcited by 173
  31. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 167
  32. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 149
  33. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 147
  34. 2023Vision-Language Models as Success Detectorscited by 138
  35. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
  36. 2023FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulationcited by 132
  37. 2026Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Groundingcited by 132
  38. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 128

50 most cited of 116 All 116 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 51.

51 papers.

Sort:

By area:

  1. 2026RLDX-1 Technical Reportcited by 25
  2. 2026Large Reward Models: Generalizable Online Robot Reward Generation with Vision-Language Modelscited by 11
  3. 2026ARM: Advantage Reward Modeling for Long-Horizon Manipulationcited by 10
  4. 2026ProcVLM: Learning Procedure-Grounded Progress Rewards for Robotic Manipulationcited by 8
  5. 2026Addressing the Orchestration Gap in Generalist Robots via Physical Agencycited by 7
  6. 2026Improving Robotic Generalist Policies via Flow Reversal Steeringcited by 5
  7. 2026Vesta: A Generalist Embodied Reasoning Modelcited by 5
  8. 2026ACE-Brain-0.5: A Unified Embodied Foundational Model for Physical Agentic AIcited by 5
  9. 2026RynnValue: Scaling Robotic Value Foundation Models with Temporal Distancecited by 5
  10. 2026RARM: Confidence-Gated Progress Reward Modeling for RL in Manipulationcited by 3
  11. 2026Progress Reward Modeling for Robotic Learning: A Comprehensive Surveycited by 3
  12. 2026RedFlow: Redirect Failure into Action-Level Corrections for Flow-matching VLA Policycited by 3
38 more
  1. 2026Rewarding DINO: Predicting Dense Rewards with Vision Foundation Modelscited by 2
  2. 2026SARM2: Multi-Task Stage Aware Reward Modeling for Self Improving Robotic Manipulationcited by 2
  3. 2026DriveJudge: Rethinking Autonomous Driving Evaluation with Vision-Language Modelscited by 2
  4. 2026PearlVLA: Progressive Embodied Action-Plan Refinement in Latent Spacecited by 2
  5. 2026VLA-FAIL: Efficient Task Failure Detection for Finetuned Vision-Language-Action Modelscited by 2
  6. 2026SAFECAST: Robust Failure Detection for VLA Policies with Contrast-Set Training and Calibrationcited by 2
  7. 2026ContactGuard: Pre-Contact Execution Monitoring with Action-Conditioned Latent World Modelscited by 2
  8. 2026PRM-as-a-Judge 1.5: A Toolkit for Robot Process Assessmentcited by 2
  9. 2026SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learningcited by 1
  10. 2026Grounded World Model for Semantically Generalizable Planningcited by 1
  11. 2026DreamAvoid: Critical-Phase Test-Time Dreaming to Avoid Failures in VLA Policiescited by 1
  12. 2026Freeform Preference Learning for Robotic Manipulationcited by 1
  13. 2026Scaling Automatic Research Agents via World Modelscited by 1
  14. 2026Reinforcing VLAs in Task-Agnostic World Modelscited by 0
  15. 2026StressDream: Steering Video World Models for Robust Policy Evaluation and Improvementcited by 0
  16. 2026Learning Process Rewards via Success Visitation Matching for Efficient RLcited by 0
  17. 2026SynCity 3000: Bootstrapping Scene-Scale 3D Diffusioncited by 0
  18. 2026Prompt-Driven Exploration: Language as an Exploration Space for VLA Reinforcement Learningcited by 0
  19. 2026Learning More from Less: Reinforcement Learning from Hindsightcited by 0
  20. 2026HELP: Human-Efficient Large-Scale Robot Post-Training with Rollout Segmentationcited by 0
  21. 2026HumanTracker: Towards Comprehensive and Human-Aligned Motion Tracking Benchmarkcited by 0
  22. 2026ConfAL-WM: Confidence-Guided Active Learning for Action-Conditioned World Modelscited by 0
  23. 2026FailBench: How Reliable are VLMs at Judging Robot Task Success?cited by 0
  24. 2026Subspace Inference Enables Efficient Active Reward Learning from Preferencescited by 0
  25. 2026Same Trajectory, Contradictory Rewards (ROBORMBENCH): Paraphrase Fragility in Vision Language Reward Modelscited by 0
  26. 2026No Free Checker: A Survey of Verifiers for Robot Policiescited by 0
  27. 2026Certified Safety Curation: Distribution-Free Guarantees for Safe Offline Reinforcement Learningcited by 0
  28. 2026PhysBrain 1.5: From Vision-Language Models to Physical Foundation Modelscited by 0
  29. 2026AnyviewMeter: Adapting Robotic Reward Models with Camera Geometry and Multi-View Attentioncited by 0
  30. 2026SynthDemo-RL: Breaking the Zero-Reward Barrier in VLA Adaptation with LLM-Guided Synthetic Demonstrationscited by 0
  31. 2026SeeQ: Training Generalist Value Functions for Long-Horizon Robotic Manipulationcited by 0
  32. 2026PF-RL: Progress Field Reinforcement Learning via Goal-Conditioned Value Geometry for Vision-Language-Action Modelscited by 0
  33. 2026Progressive Risk Estimation for Accident Anticipationcited by 0
  34. 2026Demonstration-Free Success-Probability Reward Learning for Generalist Robot Policiescited by 0
  35. 2026ARS: Agentic Reward System for Robot Learningcited by 0
  36. 2026StructRL: Online Structured Reinforcement Learning for Long-Horizon Vision-Language-Action Taskscited by 0
  37. 2026ProgressCompass: Embodied Progress Reward Models Are Lost Without the Right Contextcited by 0
  38. 2026What to Attend, What to Keep: Skill-Conditioned Visuotactile Representation with Progress-Guided Event Memorycited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Reward-DAgger: Robot-Gated Interactive Imitation Learning with General-Purpose Progress-Based Reward Models
  2. 2025RoboScape-R: Unified Reward-Observation World Models for Generalizable Robotics Training via RL
  3. 2026Robo-Dopamine 2.0: History-Conditioned and OOD-Aware Process Reward Modeling for Robotic Manipulation
  4. 2024PROGRESSOR: A Perceptually Guided Reward Estimator with Self-Supervised Online Refinement
  5. 2026TrustRoboReward: Preference-Ordered Isotonic Score Editing for Multi-Paradigm Robot Reward Models
  6. 2026CriticHack: Evaluating Visual Rewards Under Robot Policy Optimization
  7. 2019Scaling data-driven robotics with reward sketching and batch reinforcement learning
  8. 2024Learning Manipulation Skills through Robot Chain-of-Thought with Sparse Failure Guidance