mlsanityData to 14 Oct 2026

RynnValue: Scaling Robotic Value Foundation Models with Temporal Distance

arXiv 2608.09853 ↗First posted 10 Aug 2026cs.RO cs.CV cs.LGcited by 7

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

General-purpose reward models are increasingly the bottleneck for scaling robot learning, yet the recipe for learning value-related capabilities from large-scale heterogeneous corpora remains underexplored. Existing approaches tie supervision to task-internal anchors such as preferences or normalized progress, none of which transfer cleanly across embodiments and data sources. We introduce RynnValue, an open-source value foundation model for robotic manipulation that replaces these anchors with temporal distance, the directed cost-to-go from an observation to the language-specified goal. Because temporal-distance labels can be derived directly from timestamps, RynnValue scales to over 7,000 hours and roughly 3M instruction-conditioned clips without preference or progress annotations. To make temporal-value learning reliable at scale, we combine random temporal sampling, temporal-order shuffling, and value-isolation attention, suppressing shortcuts that would leave predictions insensitive to failures and regressions. Trained without preference labels, RynnValue attains an average Kendall's τa of 0.704 on RBM-EVAL-OOD, surpassing the fully preference-supervised state of the art (0.655) and more than doubling a progress-only counterpart (0.292), while generalizing zero-shot to unseen tasks, embodiments, and viewpoints. As a zero-shot reward model, RynnValue serves a range of downstream applications. Converted into dense rewards via potential-based shaping, it raises real-world policy success from 52.5% to 72.5% online and from 63.8% to 82.5% offline; used for data filtering, it improves multi-task behavior cloning success from 35.0% to 42.5%; and applied as inference-time value guidance, it lifts a frozen policy's success from 67.5% to 80.0%. These results establish temporal distance as a scalable supervision target and practical reward interface for generalist robot policies.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 51 of its 63 references that are in the map. First posted on arXiv 10 Aug 2026.

Builds on

51 of 63 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026RARM: Confidence-Gated Progress Reward Modeling for RL in Manipulation20 Jun
  2. 2026ARM: Advantage Reward Modeling for Long-Horizon Manipulation3 Apr
  3. 2026Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons2 Mar
  4. 2026TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics22 Feb
  5. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 46

  1. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  2. 2025Qwen3-VL Technical Reportcited by 2,793
  3. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  4. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
  5. 2016Generative Adversarial Imitation Learningcited by 1,597
  6. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  7. 2017Hindsight Experience Replaycited by 1,136
  8. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  9. 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulationcited by 691
  10. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 670
  11. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
  12. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 459
34 more
  1. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 406
  2. 2025AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systemscited by 405
  3. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
  4. 2016Successor Features for Transfer in Reinforcement Learningcited by 340
  5. 2025X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Modelcited by 303
  6. 2025π*0.6: a VLA That Learns From Experiencecited by 303
  7. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 260
  8. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
  9. 2025EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Videocited by 211
  10. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 194
  11. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 188
  12. 2024RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulationcited by 175
  13. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 169
  14. 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 165
  15. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 144
  16. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 134
  17. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 121
  18. 2025RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twinscited by 121
  19. 2021Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videoscited by 119
  20. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 108
  21. 2018Universal Successor Features Approximatorscited by 103
  22. 2025Galaxea Open-World Dataset and G0 Dual-System VLA Modelcited by 92
  23. 2024Vision Language Models are In-Context Value Learnerscited by 90
  24. 2018Temporal Difference Models: Model-Free Deep RL for Model-Based Controlcited by 89
  25. 2025RoboCOIN: An Open-Sourced Bimanual Robotic Data Collection for Integrated Manipulationcited by 78
  26. 2025ReWiND: Language-Guided Rewards Teach Robot Policies without New Demonstrationscited by 60
  27. 2024Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Makingcited by 56
  28. 2026RoboReward: General-Purpose Vision-Language Reward Models for Roboticscited by 53
  29. 2025InternData-A1: Pioneering High-Fidelity Synthetic Data for Pre-training Generalist Policycited by 50
  30. 2025Robo-Dopamine: General Process Reward Modeling for High-Precision Robotic Manipulationcited by 40
  31. 2026RynnBrain: Open Embodied Foundation Modelscited by 35
  32. 2024Rank2Reward: Learning Shaped Reward Functions from Passive Videocited by 32
  33. 2025TimeRewarder: Learning Dense Reward from Passive Videos via Frame-wise Temporal Distancecited by 6
  34. 2026FiRE: Enhancing MLLMs with Fine-Grained Context Learning for Complex Image Retrievalcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026UR-VC: Unsupervised Robotic Value Correction for Time-Derived Progress Proxies
  2. 2026World Value Models for Robotic Manipulation
  3. 2026SARM2: Multi-Task Stage Aware Reward Modeling for Self Improving Robotic Manipulation
  4. 2026ViVa: A Video-Generative Value Model for Robot Reinforcement Learning
  5. 2026Freeform Preference Learning for Robotic Manipulation
  6. 2026Feat2Go: Visual Feature-Grounded Value Estimation for Embodied Reinforcement Learning
  7. 2026Beyond Monotonic Progress: Retry-Supervised Value Learning for Robot Imitation
  8. 2025Training-free Generation of Temporally Consistent Rewards from VLMs