# TimeRewarder: Learning Dense Reward from Passive Videos via Frame-wise Temporal Distance

arXiv: https://arxiv.org/abs/2509.26627 · HTML: https://mlsanity.com/paper/2509.26627/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 42 of its 50 references that are in the map.
- First posted on arXiv 30 Sep 2025.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 30 of 42)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [Behavioral Cloning from Observation](https://mlsanity.com/paper/1805.01954/) (2018)
- [MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge](https://mlsanity.com/paper/2206.08853/) (2022)
- [Time-Contrastive Networks: Self-Supervised Learning from Video](https://mlsanity.com/paper/1704.06888/) (2017)
- [VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training](https://mlsanity.com/paper/2210.00030/) (2022)
- [Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning](https://mlsanity.com/paper/2107.09645/) (2021)
- [Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos](https://mlsanity.com/paper/2206.11795/) (2022)
- [ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AI](https://mlsanity.com/paper/2410.00425/) (2024)
- [How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned](https://mlsanity.com/paper/2102.02915/) (2021)
- [Generative Adversarial Imitation from Observation](https://mlsanity.com/paper/1807.06158/) (2018)
- [Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes](https://mlsanity.com/paper/2408.03539/) (2024)
- [Extreme Parkour with Legged Robots](https://mlsanity.com/paper/2309.14341/) (2023)
- [Temporal Cycle-Consistency Learning](https://mlsanity.com/paper/1904.07846/) (2019)
- [Primal Wasserstein Imitation Learning](https://mlsanity.com/paper/2006.04678/) (2020)
- [Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videos](https://mlsanity.com/paper/2103.16817/) (2021)
- [Zero-Shot Visual Imitation](https://mlsanity.com/paper/1804.08606/) (2018)
- [Imitating Latent Policies from Observation](https://mlsanity.com/paper/1805.07914/) (2018)
- [State-Only Imitation Learning for Dexterous Manipulation](https://mlsanity.com/paper/2004.04650/) (2020)
- [Combining Self-Supervised Learning and Imitation for Vision-Based Rope Manipulation](https://mlsanity.com/paper/1703.02018/) (2017)
- [Vision Language Models are In-Context Value Learners](https://mlsanity.com/paper/2411.04549/) (2024)
- [Video Prediction Models as Rewards for Reinforcement Learning](https://mlsanity.com/paper/2305.14343/) (2023)
- [Watch and Match: Supercharging Imitation with Regularized Optimal Transport](https://mlsanity.com/paper/2206.15469/) (2022)
- [Diffusion Reward: Learning Rewards via Conditional Video Diffusion](https://mlsanity.com/paper/2312.14134/) (2023)
- [Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreement](https://mlsanity.com/paper/1910.04417/) (2019)

## Built on by (most cited first, 6 of 6)

- [RynnValue: Scaling Robotic Value Foundation Models with Temporal Distance](https://mlsanity.com/paper/2608.09853/) (2026)
- [SHIFT: Motion Alignment in Video Diffusion Models with Adversarial Hybrid Fine-Tuning](https://mlsanity.com/paper/2603.17426/) (2026)
- [RARM: Confidence-Gated Progress Reward Modeling for RL in Manipulation](https://mlsanity.com/paper/2606.22027/) (2026)
- [Reward Prediction with Factorized World States](https://mlsanity.com/paper/2603.09400/) (2026)
- [ARS: Agentic Reward System for Robot Learning](https://mlsanity.com/paper/2609.34484/) (2026)
- [Disentangling Task-Relevant Latent Dynamics for Reward Modeling](https://mlsanity.com/paper/2610.14159/) (2026)

## Abstract (arXiv)

Designing dense rewards is crucial for reinforcement learning (RL), yet in robotics it often demands extensive manual effort and lacks scalability. One promising solution is to view task progress as a dense reward signal, as it quantifies the degree to which actions advance the system toward task completion over time. We present TimeRewarder, a simple yet effective reward learning method that derives progress estimation signals from passive videos, including robot demonstrations and human videos, by modeling temporal distances between frame pairs. We then demonstrate how TimeRewarder can supply step-wise proxy rewards to guide reinforcement learning. In our comprehensive experiments on ten challenging Meta-World tasks, we show that TimeRewarder dramatically improves RL for sparse-reward tasks, achieving nearly perfect success in 9/10 tasks with only 200,000 environment interactions per task. This approach outperformed previous methods and even the manually designed environment dense reward on both the final success rate and sample efficiency. Moreover, we show that TimeRewarder pretraining can exploit real-world human videos, highlighting its potential as a scalable approach to rich reward signals from diverse video sources.