RynnValue: Scaling Robotic Value Foundation Models with Temporal Distance
On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF
Abstract (arXiv)
General-purpose reward models are increasingly the bottleneck for scaling robot learning, yet the recipe for learning value-related capabilities from large-scale heterogeneous corpora remains underexplored. Existing approaches tie supervision to task-internal anchors such as preferences or normalized progress, none of which transfer cleanly across embodiments and data sources. We introduce RynnValue, an open-source value foundation model for robotic manipulation that replaces these anchors with temporal distance, the directed cost-to-go from an observation to the language-specified goal. Because temporal-distance labels can be derived directly from timestamps, RynnValue scales to over 7,000 hours and roughly 3M instruction-conditioned clips without preference or progress annotations. To make temporal-value learning reliable at scale, we combine random temporal sampling, temporal-order shuffling, and value-isolation attention, suppressing shortcuts that would leave predictions insensitive to failures and regressions. Trained without preference labels, RynnValue attains an average Kendall's τa of 0.704 on RBM-EVAL-OOD, surpassing the fully preference-supervised state of the art (0.655) and more than doubling a progress-only counterpart (0.292), while generalizing zero-shot to unseen tasks, embodiments, and viewpoints. As a zero-shot reward model, RynnValue serves a range of downstream applications. Converted into dense rewards via potential-based shaping, it raises real-world policy success from 52.5% to 72.5% online and from 63.8% to 82.5% offline; used for data filtering, it improves multi-task behavior cloning success from 35.0% to 42.5%; and applied as inference-time value guidance, it lifts a frozen policy's success from 67.5% to 80.0%. These results establish temporal distance as a scalable supervision target and practical reward interface for generalist robot policies.
Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 51 of its 63 references that are in the map. First posted on arXiv 10 Aug 2026.
Builds on
51 of 63 references.
Closest ancestors · 5
Same direction, newest first
- 2026RARM: Confidence-Gated Progress Reward Modeling for RL in Manipulation20 Jun
- 2026ARM: Advantage Reward Modeling for Long-Horizon Manipulation3 Apr
- 2026Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons2 Mar
- 2026TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics22 Feb
- 2017Deep reinforcement learning from human preferences12 Jun
Other references · 46
- 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
- 2025Qwen3-VL Technical Reportcited by 2,793
- 2020Shortcut Learning in Deep Neural Networkscited by 1,969
- 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
- 2016Generative Adversarial Imitation Learningcited by 1,597
- 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
- 2017Hindsight Experience Replaycited by 1,136
- 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
- 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulationcited by 691
- 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 670
- 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
- 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 459
34 more
- 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 406
- 2025AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systemscited by 405
- 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
- 2016Successor Features for Transfer in Reinforcement Learningcited by 340
- 2025X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Modelcited by 303
- 2025π*0.6: a VLA That Learns From Experiencecited by 303
- 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 260
- 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
- 2025EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Videocited by 211
- 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 194
- 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 188
- 2024RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulationcited by 175
- 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 169
- 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 165
- 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 144
- 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 134
- 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 121
- 2025RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twinscited by 121
- 2021Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videoscited by 119
- 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 108
- 2018Universal Successor Features Approximatorscited by 103
- 2025Galaxea Open-World Dataset and G0 Dual-System VLA Modelcited by 92
- 2024Vision Language Models are In-Context Value Learnerscited by 90
- 2018Temporal Difference Models: Model-Free Deep RL for Model-Based Controlcited by 89
- 2025RoboCOIN: An Open-Sourced Bimanual Robotic Data Collection for Integrated Manipulationcited by 78
- 2025ReWiND: Language-Guided Rewards Teach Robot Policies without New Demonstrationscited by 60
- 2024Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Makingcited by 56
- 2026RoboReward: General-Purpose Vision-Language Reward Models for Roboticscited by 53
- 2025InternData-A1: Pioneering High-Fidelity Synthetic Data for Pre-training Generalist Policycited by 50
- 2025Robo-Dopamine: General Process Reward Modeling for High-Precision Robotic Manipulationcited by 40
- 2026RynnBrain: Open Embodied Foundation Modelscited by 35
- 2024Rank2Reward: Learning Shaped Reward Functions from Passive Videocited by 32
- 2025TimeRewarder: Learning Dense Reward from Passive Videos via Frame-wise Temporal Distancecited by 6
- 2026FiRE: Enhancing MLLMs with Fine-Grained Context Learning for Complex Image Retrievalcited by 1
Built on by
Citing papers per year: 2026: 7.
7 papers.
- 2026PRM-as-a-Judge 1.5: A Toolkit for Robot Process Assessmentcited by 3
- 2026AnyviewMeter: Adapting Robotic Reward Models with Camera Geometry and Multi-View Attentioncited by 0
- 2026ProTracer: Proprioception-Guided Failure Diagnosis in Robot Manipulationcited by 0
- 2026SeeQ: Training Generalist Value Functions for Long-Horizon Robotic Manipulationcited by 0
- 2026Demonstration-Free Success-Probability Reward Learning for Generalist Robot Policiescited by 0
- 2026Reward-DAgger: Robot-Gated Interactive Imitation Learning with General-Purpose Progress-Based Reward Modelscited by 0
- 2026VeriFine: Scaling Verification for Self-Improvement in Embodied Reasoningcited by 0
- 2026VeriFine: Scaling Verification for Self-Improvement in Embodied Reasoning6 Oct 2026
- 2026Reward-DAgger: Robot-Gated Interactive Imitation Learning with General-Purpose Progress-Based Reward Models2 Oct 2026
- 2026Demonstration-Free Success-Probability Reward Learning for Generalist Robot Policies27 Sep 2026
- 2026SeeQ: Training Generalist Value Functions for Long-Horizon Robotic Manipulation18 Sep 2026
- 2026ProTracer: Proprioception-Guided Failure Diagnosis in Robot Manipulation18 Sep 2026
- 2026AnyviewMeter: Adapting Robotic Reward Models with Camera Geometry and Multi-View Attention17 Sep 2026
- 2026PRM-as-a-Judge 1.5: A Toolkit for Robot Process Assessment14 Aug 2026
Similar, not cited
Close in wording, no citation either way
- 2026UR-VC: Unsupervised Robotic Value Correction for Time-Derived Progress Proxies
- 2026World Value Models for Robotic Manipulation
- 2026SARM2: Multi-Task Stage Aware Reward Modeling for Self Improving Robotic Manipulation
- 2026ViVa: A Video-Generative Value Model for Robot Reinforcement Learning
- 2026Freeform Preference Learning for Robotic Manipulation
- 2026Feat2Go: Visual Feature-Grounded Value Estimation for Embodied Reinforcement Learning
- 2026Beyond Monotonic Progress: Retry-Supervised Value Learning for Robot Imitation
- 2025Training-free Generation of Temporally Consistent Rewards from VLMs