# Reinforcement Learning from Passive Data via Latent Intentions

arXiv: https://arxiv.org/abs/2304.04782 · HTML: https://mlsanity.com/paper/2304.04782/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 42 references that are in the map.
- First posted on arXiv 10 Apr 2023.

## Builds on (most cited first, 30 of 32)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [R3M: A Universal Visual Representation for Robot Manipulation](https://mlsanity.com/paper/2203.12601/) (2022)
- [Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels](https://mlsanity.com/paper/2004.13649/) (2020)
- [Behavioral Cloning from Observation](https://mlsanity.com/paper/1805.01954/) (2018)
- [Time-Contrastive Networks: Self-Supervised Learning from Video](https://mlsanity.com/paper/1704.06888/) (2017)
- [VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training](https://mlsanity.com/paper/2210.00030/) (2022)
- [Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos](https://mlsanity.com/paper/2206.11795/) (2022)
- [Successor Features for Transfer in Reinforcement Learning](https://mlsanity.com/paper/1606.05312/) (2016)
- [Contrastive Learning as Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2206.07568/) (2022)
- [Visual Reinforcement Learning with Imagined Goals](https://mlsanity.com/paper/1807.04742/) (2018)
- [Masked Visual Pre-training for Motor Control](https://mlsanity.com/paper/2203.06173/) (2022)
- [Generative Adversarial Imitation from Observation](https://mlsanity.com/paper/1807.06158/) (2018)
- [Masked World Models for Visual Control](https://mlsanity.com/paper/2206.14244/) (2022)
- [Learning One Representation to Optimize All Rewards](https://mlsanity.com/paper/2103.07945/) (2021)
- [Reinforcement Learning with Action-Free Pre-Training from Videos](https://mlsanity.com/paper/2203.13880/) (2022)
- [XIRL: Cross-embodiment Inverse Reinforcement Learning](https://mlsanity.com/paper/2106.03911/) (2021)
- [Third-Person Imitation Learning](https://mlsanity.com/paper/1703.01703/) (2017)
- [Universal Successor Features Approximators](https://mlsanity.com/paper/1812.07626/) (2018)
- [Eigenoption Discovery through the Deep Successor Representation](https://mlsanity.com/paper/1710.11089/) (2017)
- [Reinforcement Learning with Videos: Combining Offline Observations with Interaction](https://mlsanity.com/paper/2011.06507/) (2020)
- [Deep Successor Reinforcement Learning](https://mlsanity.com/paper/1606.02396/) (2016)
- [You Can't Count on Luck: Why Decision Transformers and RvS Fail in Stochastic Environments](https://mlsanity.com/paper/2205.15967/) (2022)
- [Dichotomy of Control: Separating What You Can Control from What You Cannot](https://mlsanity.com/paper/2210.13435/) (2022)
- [The MAGICAL Benchmark for Robust Imitation](https://mlsanity.com/paper/2011.00401/) (2020)
- [Addressing Optimism Bias in Sequence Modeling for Reinforcement Learning](https://mlsanity.com/paper/2207.10295/) (2022)

## Built on by (most cited first, 30 of 68)

- [OGBench: Benchmarking Offline Goal-Conditioned RL](https://mlsanity.com/paper/2410.20092/) (2024)
- [HIQL: Offline Goal-Conditioned RL with Latent States as Actions](https://mlsanity.com/paper/2307.11949/) (2023)
- [Learning to Act without Actions](https://mlsanity.com/paper/2312.10812/) (2023)
- [Foundation Policies with Hilbert Representations](https://mlsanity.com/paper/2402.15567/) (2024)
- [Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making](https://mlsanity.com/paper/2406.17098/) (2024)
- [Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Models](https://mlsanity.com/paper/2504.11054/) (2025)
- [Latent Action Learning Requires Supervision in the Presence of Distractors](https://mlsanity.com/paper/2502.00379/) (2025)
- [Towards Generalist Robot Learning from Internet Video: A Survey](https://mlsanity.com/paper/2404.19664/) (2024)
- [Robotic Offline RL from Internet Videos via Value-Function Pre-Training](https://mlsanity.com/paper/2309.13041/) (2023)
- [Multi-agent Reinforcement Learning: A Comprehensive Survey](https://mlsanity.com/paper/2312.10256/) (2023)
- [Inverse Dynamics Pretraining Learns Good Representations for Multitask Imitation](https://mlsanity.com/paper/2305.16985/) (2023)
- [Temporal Difference Flows](https://mlsanity.com/paper/2503.09817/) (2025)
- [TD-JEPA: Latent-predictive Representations for Zero-Shot Reinforcement Learning](https://mlsanity.com/paper/2510.00739/) (2025)
- [Accelerating Exploration with Unlabeled Prior Data](https://mlsanity.com/paper/2311.05067/) (2023)
- [Value-guided action planning with JEPA world models](https://mlsanity.com/paper/2601.00844/) (2025)
- [Proto Successor Measure: Representing the Behavior Space of an RL Agent](https://mlsanity.com/paper/2411.19418/) (2024)
- [AMPLIFY: Actionless Motion Priors for Robot Learning from Videos](https://mlsanity.com/paper/2506.14198/) (2025)
- [A Distributional Analogue to the Successor Representation](https://mlsanity.com/paper/2402.08530/) (2024)
- [Dual Goal Representations](https://mlsanity.com/paper/2510.06714/) (2025)
- [Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2509.06782/) (2025)
- [Unsupervised Behavior Extraction via Random Intent Priors](https://mlsanity.com/paper/2310.18687/) (2023)
- [Diffusion Imitation from Observation](https://mlsanity.com/paper/2410.05429/) (2024)
- [Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration](https://mlsanity.com/paper/2410.18076/) (2024)
- [Transitive RL: Value Learning via Divide and Conquer](https://mlsanity.com/paper/2510.22512/) (2025)
- [Unsupervised-to-Online Reinforcement Learning](https://mlsanity.com/paper/2408.14785/) (2024)
- [Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matching](https://mlsanity.com/paper/2411.07007/) (2024)
- [PEAC: Unsupervised Pre-training for Cross-Embodiment Reinforcement Learning](https://mlsanity.com/paper/2405.14073/) (2024)
- [Object-Centric Latent Action Learning](https://mlsanity.com/paper/2502.09680/) (2025)
- [Intention-Conditioned Flow Occupancy Models](https://mlsanity.com/paper/2506.08902/) (2025)
- [Align Your Intents: Offline Imitation Learning via Optimal Transport](https://mlsanity.com/paper/2402.13037/) (2024)

## Abstract (arXiv)

Passive observational data, such as human videos, is abundant and rich in information, yet remains largely untapped by current RL methods. Perhaps surprisingly, we show that passive data, despite not having reward or action labels, can still be used to learn features that accelerate downstream RL. Our approach learns from passive data by modeling intentions: measuring how the likelihood of future outcomes change when the agent acts to achieve a particular task. We propose a temporal difference learning objective to learn about intentions, resulting in an algorithm similar to conventional RL, but which learns entirely from passive data. When optimizing this objective, our agent simultaneously learns representations of states, of policies, and of possible outcomes in an environment, all from raw observational data. Both theoretically and empirically, this scheme learns features amenable for value prediction for downstream tasks, and our experiments demonstrate the ability to learn from many forms of passive data, including cross-embodiment video data and YouTube videos.