# Reinforcement Learning with Videos: Combining Offline Observations with Interaction

arXiv: https://arxiv.org/abs/2011.06507 · HTML: https://mlsanity.com/paper/2011.06507/

## Key facts

- Cited by 82 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Nov 2020.

## Built on by (most cited first, 30 of 82)

- [R3M: A Universal Visual Representation for Robot Manipulation](https://mlsanity.com/paper/2203.12601/) (2022)
- [VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training](https://mlsanity.com/paper/2210.00030/) (2022)
- [MimicPlay: Long-Horizon Imitation Learning by Watching Human Play](https://mlsanity.com/paper/2302.12422/) (2023)
- [DexMV: Imitation Learning for Dexterous Manipulation from Human Videos](https://mlsanity.com/paper/2108.05877/) (2021)
- [A Survey of Zero-shot Generalisation in Deep Reinforcement Learning](https://mlsanity.com/paper/2111.09794/) (2021)
- [Learning to Act from Actionless Videos through Dense Correspondences](https://mlsanity.com/paper/2310.08576/) (2023)
- [HIQL: Offline Goal-Conditioned RL with Latent States as Actions](https://mlsanity.com/paper/2307.11949/) (2023)
- [Flow as the Cross-Domain Manipulation Interface](https://mlsanity.com/paper/2407.15208/) (2024)
- [Learning to Act without Actions](https://mlsanity.com/paper/2312.10812/) (2023)
- [Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videos](https://mlsanity.com/paper/2103.16817/) (2021)
- [Reinforcement Learning with Action-Free Pre-Training from Videos](https://mlsanity.com/paper/2203.13880/) (2022)
- [XIRL: Cross-embodiment Inverse Reinforcement Learning](https://mlsanity.com/paper/2106.03911/) (2021)
- [VideoDex: Learning Dexterity from Internet Videos](https://mlsanity.com/paper/2212.04498/) (2022)
- [Learning by Watching: Physical Imitation of Manipulation Skills from Human Videos](https://mlsanity.com/paper/2101.07241/) (2021)
- [From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperation](https://mlsanity.com/paper/2204.12490/) (2022)
- [Video Prediction Models as Rewards for Reinforcement Learning](https://mlsanity.com/paper/2305.14343/) (2023)
- [XSkill: Cross Embodiment Skill Discovery](https://mlsanity.com/paper/2307.09955/) (2023)
- [Reinforcement Learning from Passive Data via Latent Intentions](https://mlsanity.com/paper/2304.04782/) (2023)
- [Human Hands as Probes for Interactive Object Understanding](https://mlsanity.com/paper/2112.09120/) (2021)
- [Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformers](https://mlsanity.com/paper/2403.12943/) (2024)
- [Learning Generalizable Dexterous Manipulation from Human Grasp Affordance](https://mlsanity.com/paper/2204.02320/) (2022)
- [Zero-Shot Robot Manipulation from Passive Human Videos](https://mlsanity.com/paper/2302.02011/) (2023)
- [Graph Inverse Reinforcement Learning from Diverse Videos](https://mlsanity.com/paper/2207.14299/) (2022)
- [MobILE: Model-Based Imitation Learning From Observation Alone](https://mlsanity.com/paper/2102.10769/) (2021)
- [Latent Action Learning Requires Supervision in the Presence of Distractors](https://mlsanity.com/paper/2502.00379/) (2025)
- [Towards Generalist Robot Learning from Internet Video: A Survey](https://mlsanity.com/paper/2404.19664/) (2024)
- [ViViDex: Learning Vision-based Dexterous Manipulation from Human Videos](https://mlsanity.com/paper/2404.15709/) (2024)
- [Learning Continuous Grasping Function with a Dexterous Hand from Human Demonstrations](https://mlsanity.com/paper/2207.05053/) (2022)
- [Robotic Offline RL from Internet Videos via Value-Function Pre-Training](https://mlsanity.com/paper/2309.13041/) (2023)
- [Rank2Reward: Learning Shaped Reward Functions from Passive Video](https://mlsanity.com/paper/2404.14735/) (2024)

## Abstract (arXiv)

Reinforcement learning is a powerful framework for robots to acquire skills from experience, but often requires a substantial amount of online data collection. As a result, it is difficult to collect sufficiently diverse experiences that are needed for robots to generalize broadly. Videos of humans, on the other hand, are a readily available source of broad and interesting experiences. In this paper, we consider the question: can we perform reinforcement learning directly on experience collected by humans? This problem is particularly difficult, as such videos are not annotated with actions and exhibit substantial visual domain shift relative to the robot's embodiment. To address these challenges, we propose a framework for reinforcement learning with videos (RLV). RLV learns a policy and value function using experience collected by humans in combination with data collected by robots. In our experiments, we find that RLV is able to leverage such videos to learn challenging vision-based skills with less than half as many samples as RL methods that learn from scratch.