mlsanityData to 14 Oct 2026

Disentangling Task-Relevant Latent Dynamics for Reward Modeling

arXiv 2610.14159 ↗First posted 9 Oct 2026cs.RO cs.CVcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF also fits: Reward hacking and misspecification

also in 1 more
Abstract (arXiv)

Reward design remains a major bottleneck for reinforcement learning in robotic manipulation. Handcrafted dense rewards can compensate for the limited guidance provided by sparse rewards, but require substantial task-specific reward engineering. Learning rewards from videos provides a scalable, data-driven alternative to manual reward design. However, existing video-based rewards are often computed in entangled visual spaces where task-relevant state changes are confounded with extraneous visual variation. As a result, the learned reward may respond to visual distractors instead of actual task execution. To address this limitation, we propose DiTaR (Disentangling Task-Relevant Latent Dynamics for Reward Modeling), a framework for learning dense rewards from heterogeneous videos. DiTaR learns a continuous task-relevant representation from multi-frame video latents through an asymmetric two-stage procedure that suppresses task-irrelevant visual variation. In the task-relevant latent space, we train a history- and language-conditioned rectified-flow model to capture task dynamics and derive a directional dense reward from temporal changes in dynamics compatibility. Together, these components enable reward modeling directly over task-relevant dynamics rather than generic visual variation. DiTaR substantially improves policy learning on MetaWorld-v3 and ManiSkill over video-based reward baselines, and transfers to real-world offline RL through reward relabeling. Reward analyses further show that DiTaR provides more reliable rewards.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 56 of its 57 references that are in the map. First posted on arXiv 9 Oct 2026.

Builds on

56 of 57 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Video-Language Critic: Transferable Reward Functions for Language-Conditioned Robotics30 May
  2. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning19 Oct

Other references · 54

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  3. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  4. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
  5. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  6. 2023Mastering Diverse Domains through World Modelscited by 1,253
  7. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  8. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  9. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 852
  10. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 809
  11. 2024Genie: Generative Interactive Environmentscited by 747
  12. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
38 more
  1. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 657
  2. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 531
  3. 2023Learning Universal Policies via Text-Guided Video Generationcited by 522
  4. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  5. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 459
  6. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 410
  7. 2025AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systemscited by 405
  8. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
  9. 2024DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planningcited by 386
  10. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  11. 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 327
  12. 2024Latent Action Pretraining from Videoscited by 321
  13. 2026LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixelscited by 273
  14. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 188
  15. 2023Language-Driven Representation Learning for Roboticscited by 165
  16. 2019Temporal Cycle-Consistency Learningcited by 154
  17. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 134
  18. 2024Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulationcited by 124
  19. 2021Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videoscited by 119
  20. 2026V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learningcited by 117
  21. 2018Playing hard exploration games by watching YouTubecited by 99
  22. 2024Vision Language Models are In-Context Value Learnerscited by 90
  23. 2023Video Prediction Models as Rewards for Reinforcement Learningcited by 78
  24. 2025A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learningcited by 71
  25. 2025ReWiND: Language-Guided Rewards Teach Robot Policies without New Demonstrationscited by 60
  26. 2023Diffusion Reward: Learning Rewards via Conditional Video Diffusioncited by 57
  27. 2026RoboReward: General-Purpose Vision-Language Reward Models for Roboticscited by 53
  28. 2026LDA-1B: Scaling Latent Dynamics Action Model via Universal Embodied Data Ingestioncited by 52
  29. 2025Meta-World+: An Improved, Standardized, RL Benchmarkcited by 43
  30. 2025SARM: Stage-Aware Reward Modeling for Long Horizon Robot Manipulationcited by 42
  31. 2024Rank2Reward: Learning Shaped Reward Functions from Passive Videocited by 32
  32. 2026CLAP: Contrastive Latent Action Pretraining for Learning Vision-Language-Action Models from Human Videoscited by 31
  33. 2024Grounding Video Models to Actions through Goal Conditioned Explorationcited by 26
  34. 2022Learning Reward Functions for Robotic Manipulation by Observing Humanscited by 17
  35. 2024PROGRESSOR: A Perceptually Guided Reward Estimator with Self-Supervised Online Refinementcited by 13
  36. 2024Adapt2Reward: Adapting Video-Language Models to Generalizable Robotic Rewards via Failure Promptscited by 10
  37. 2025Subtask-Aware Visual Reward Learning from Segmented Demonstrationscited by 10
  38. 2026GuidedVLA: Specifying Task-Relevant Factors via Plug-and-Play Action Attention Specializationcited by 10

50 most cited of 54 All 54 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Rewarding DINO: Predicting Dense Rewards with Vision Foundation Models
  2. 2026GORDON: Graph-based Object-centric Rewards for Decomposition of Long-Horizon Manipulation
  3. 2026TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics
  4. 2026Beyond Pixels: Learning Invariant Rewards for Real-World Robotics From a Few Demonstrations
  5. 2025Robo-Dopamine: General Process Reward Modeling for High-Precision Robotic Manipulation
  6. 2026RARM: Confidence-Gated Progress Reward Modeling for RL in Manipulation
  7. 2026Stage-Transition Dense Reward Modeling for Reinforcement Learning
  8. 2026AnyviewMeter: Adapting Robotic Reward Models with Camera Geometry and Multi-View Attention