# PROGRESSOR: A Perceptually Guided Reward Estimator with Self-Supervised Online Refinement

arXiv: https://arxiv.org/abs/2411.17764 · HTML: https://mlsanity.com/paper/2411.17764/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 33 of its 48 references that are in the map.
- First posted on arXiv 26 Nov 2024.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning](https://mlsanity.com/paper/1910.00177/) (2019)
- [R3M: A Universal Visual Representation for Robot Manipulation](https://mlsanity.com/paper/2203.12601/) (2022)
- [Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels](https://mlsanity.com/paper/2004.13649/) (2020)
- [Rescaling Egocentric Vision](https://mlsanity.com/paper/2006.13256/) (2020)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Time-Contrastive Networks: Self-Supervised Learning from Video](https://mlsanity.com/paper/1704.06888/) (2017)
- [Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization](https://mlsanity.com/paper/1603.00448/) (2016)
- [VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training](https://mlsanity.com/paper/2210.00030/) (2022)
- [Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos](https://mlsanity.com/paper/2206.11795/) (2022)
- [Contrastive Learning as Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2206.07568/) (2022)
- [Inverse Reward Design](https://mlsanity.com/paper/1711.02827/) (2017)
- [DexMV: Imitation Learning for Dexterous Manipulation from Human Videos](https://mlsanity.com/paper/2108.05877/) (2021)
- [Maximum Entropy Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/1507.04888/) (2015)
- [Human-to-Robot Imitation in the Wild](https://mlsanity.com/paper/2207.09450/) (2022)
- [Imitation Learning as f-Divergence Minimization](https://mlsanity.com/paper/1905.12888/) (2019)
- [State-Only Imitation Learning for Dexterous Manipulation](https://mlsanity.com/paper/2004.04650/) (2020)
- [Representation Matters: Offline Pretraining for Sequential Decision Making](https://mlsanity.com/paper/2102.05815/) (2021)
- [Vision Language Models are In-Context Value Learners](https://mlsanity.com/paper/2411.04549/) (2024)
- [Video Prediction Models as Rewards for Reinforcement Learning](https://mlsanity.com/paper/2305.14343/) (2023)
- [Watch and Match: Supercharging Imitation with Regularized Optimal Transport](https://mlsanity.com/paper/2206.15469/) (2022)
- [Diffusion Reward: Learning Rewards via Conditional Video Diffusion](https://mlsanity.com/paper/2312.14134/) (2023)
- [Learning to Drive by Watching YouTube Videos: Action-Conditioned Contrastive Policy Pretraining](https://mlsanity.com/paper/2204.02393/) (2022)

## Built on by (most cited first, 13 of 13)

- [ZeroMimic: Distilling Robotic Manipulation Skills from Web Videos](https://mlsanity.com/paper/2503.23877/) (2025)
- [Large Reward Models: Generalizable Online Robot Reward Generation with Vision-Language Models](https://mlsanity.com/paper/2603.16065/) (2026)
- [TimeRewarder: Learning Dense Reward from Passive Videos via Frame-wise Temporal Distance](https://mlsanity.com/paper/2509.26627/) (2025)
- [RARM: Confidence-Gated Progress Reward Modeling for RL in Manipulation](https://mlsanity.com/paper/2606.22027/) (2026)
- [Bi-HIL: Bilateral Control-Based Multimodal Hierarchical Imitation Learning via Subtask-Level Progress Rate and Keyframe Memory for Long-Horizon Contact-Rich Robotic Manipulation](https://mlsanity.com/paper/2603.13315/) (2026)
- [MVR: Multi-view Video Reward Shaping for Reinforcement Learning](https://mlsanity.com/paper/2603.01694/) (2026)
- [A Hierarchical Spatiotemporal Action Tokenizer for In-Context Imitation Learning in Robotics](https://mlsanity.com/paper/2604.15215/) (2026)
- [Beyond Pixels: Learning Invariant Rewards for Real-World Robotics From a Few Demonstrations](https://mlsanity.com/paper/2605.22123/) (2026)
- [StageGuard: Learning Stage Transitions for Long-Horizon Robot Tasks via Agentic Distillation](https://mlsanity.com/paper/2609.20791/) (2026)
- [Completion at the Boundary (CaB): Deployable Switching with Completion-Aware Control under Limited Calibration](https://mlsanity.com/paper/2606.00145/) (2026)
- [PHIRL: Aligning Learned Rewards with Task Progress for Inverse Reinforcement Learning](https://mlsanity.com/paper/2609.31855/) (2026)
- [ARS: Agentic Reward System for Robot Learning](https://mlsanity.com/paper/2609.34484/) (2026)
- [Disentangling Task-Relevant Latent Dynamics for Reward Modeling](https://mlsanity.com/paper/2610.14159/) (2026)

## Abstract (arXiv)

We present PROGRESSOR, a novel framework that learns a task-agnostic reward function from videos, enabling policy training through goal-conditioned reinforcement learning (RL) without manual supervision. Underlying this reward is an estimate of the distribution over task progress as a function of the current, initial, and goal observations that is learned in a self-supervised fashion. Crucially, PROGRESSOR refines rewards adversarially during online RL training by pushing back predictions for out-of-distribution observations, to mitigate distribution shift inherent in non-expert observations. Utilizing this progress prediction as a dense reward together with an adversarial push-back, we show that PROGRESSOR enables robots to learn complex behaviors without any external supervision. Pretrained on large-scale egocentric human video from EPIC-KITCHENS, PROGRESSOR requires no fine-tuning on in-domain task-specific data for generalization to real-robot offline RL under noisy demonstrations, outperforming contemporary methods that provide dense visual reward for robotic learning. Our findings highlight the potential of PROGRESSOR for scalable robotic applications where direct action labels and task-specific rewards are not readily available.