# Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videos

arXiv: https://arxiv.org/abs/2103.16817 · HTML: https://mlsanity.com/paper/2103.16817/

## Key facts

- Cited by 119 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 61 references that are in the map.
- First posted on arXiv 31 Mar 2021.

## Builds on (most cited first, 30 of 43)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [Time-Contrastive Networks: Self-Supervised Learning from Video](https://mlsanity.com/paper/1704.06888/) (2017)
- [Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization](https://mlsanity.com/paper/1603.00448/) (2016)
- [Deep Visual Foresight for Planning Robot Motion](https://mlsanity.com/paper/1610.00696/) (2016)
- [Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images](https://mlsanity.com/paper/1506.07365/) (2015)
- [RoboNet: Large-Scale Multi-Robot Learning](https://mlsanity.com/paper/1910.11215/) (2019)
- [Visual Reinforcement Learning with Imagined Goals](https://mlsanity.com/paper/1807.04742/) (2018)
- [Stochastic Variational Video Prediction](https://mlsanity.com/paper/1710.11252/) (2017)
- [Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control](https://mlsanity.com/paper/1812.00568/) (2018)
- [Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours](https://mlsanity.com/paper/1509.06825/) (2015)
- [RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation](https://mlsanity.com/paper/1811.02790/) (2018)
- [Maximum Entropy Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/1507.04888/) (2015)
- [Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translation](https://mlsanity.com/paper/1707.03374/) (2017)
- [One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning](https://mlsanity.com/paper/1802.01557/) (2018)
- [Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learning](https://mlsanity.com/paper/1803.09956/) (2018)
- [End-to-End Robotic Reinforcement Learning without Reward Engineering](https://mlsanity.com/paper/1904.07854/) (2019)
- [The Ingredients of Real-World Robotic Reinforcement Learning](https://mlsanity.com/paper/2004.12570/) (2020)
- [Visual Imitation Made Easy](https://mlsanity.com/paper/2008.04899/) (2020)
- [AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos](https://mlsanity.com/paper/1912.04443/) (2019)
- [Learning by Watching: Physical Imitation of Manipulation Skills from Human Videos](https://mlsanity.com/paper/2101.07241/) (2021)
- [Reinforcement Learning with Videos: Combining Offline Observations with Interaction](https://mlsanity.com/paper/2011.06507/) (2020)
- [Robot Learning in Homes: Improving Generalization and Reducing Dataset Bias](https://mlsanity.com/paper/1807.07049/) (2018)
- [Third-Person Visual Imitation Learning via Decoupled Hierarchical Controller](https://mlsanity.com/paper/1911.09676/) (2019)
- [Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definition](https://mlsanity.com/paper/1805.11686/) (2018)

## Built on by (most cited first, 30 of 119)

- [A Generalist Agent](https://mlsanity.com/paper/2205.06175/) (2022)
- [R3M: A Universal Visual Representation for Robot Manipulation](https://mlsanity.com/paper/2203.12601/) (2022)
- [MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge](https://mlsanity.com/paper/2206.08853/) (2022)
- [VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training](https://mlsanity.com/paper/2210.00030/) (2022)
- [Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation](https://mlsanity.com/paper/2401.02117/) (2024)
- [π*0.6: a VLA That Learns From Experience](https://mlsanity.com/paper/2511.14759/) (2025)
- [Reward Design with Language Models](https://mlsanity.com/paper/2303.00001/) (2023)
- [Affordances from Human Videos as a Versatile Representation for Robotics](https://mlsanity.com/paper/2304.08488/) (2023)
- [MimicPlay: Long-Horizon Imitation Learning by Watching Human Play](https://mlsanity.com/paper/2302.12422/) (2023)
- [HumanPlus: Humanoid Shadowing and Imitation from Humans](https://mlsanity.com/paper/2406.10454/) (2024)
- [A Survey of Zero-shot Generalisation in Deep Reinforcement Learning](https://mlsanity.com/paper/2111.09794/) (2021)
- [Language-Driven Representation Learning for Robotics](https://mlsanity.com/paper/2302.12766/) (2023)
- [Learning to Act from Actionless Videos through Dense Correspondences](https://mlsanity.com/paper/2310.08576/) (2023)
- [Human-to-Robot Imitation in the Wild](https://mlsanity.com/paper/2207.09450/) (2022)
- [Structured World Models from Human Videos](https://mlsanity.com/paper/2308.10901/) (2023)
- [Data Scaling Laws in Imitation Learning for Robotic Manipulation](https://mlsanity.com/paper/2410.18647/) (2024)
- [Vision-Language Models as Success Detectors](https://mlsanity.com/paper/2303.07280/) (2023)
- [RoboCLIP: One Demonstration is Enough to Learn Robot Policies](https://mlsanity.com/paper/2310.07899/) (2023)
- [Flow as the Cross-Domain Manipulation Interface](https://mlsanity.com/paper/2407.15208/) (2024)
- [Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation](https://mlsanity.com/paper/2109.01115/) (2021)
- [Reinforcement Learning with Action-Free Pre-Training from Videos](https://mlsanity.com/paper/2203.13880/) (2022)
- [DALL-E-Bot: Introducing Web-Scale Diffusion Models to Robotics](https://mlsanity.com/paper/2210.02438/) (2022)
- [VideoDex: Learning Dexterity from Internet Videos](https://mlsanity.com/paper/2212.04498/) (2022)
- [Deep Hierarchical Planning from Pixels](https://mlsanity.com/paper/2206.04114/) (2022)
- [Instruction-driven history-aware policies for robotic manipulations](https://mlsanity.com/paper/2209.04899/) (2022)
- [Vision Language Models are In-Context Value Learners](https://mlsanity.com/paper/2411.04549/) (2024)
- [Dreamitate: Real-World Visuomotor Policy Learning via Video Generation](https://mlsanity.com/paper/2406.16862/) (2024)
- [Humanoid Policy ~ Human Policy](https://mlsanity.com/paper/2503.13441/) (2025)
- [XSkill: Cross Embodiment Skill Discovery](https://mlsanity.com/paper/2307.09955/) (2023)
- [Conservative Data Sharing for Multi-Task Offline Reinforcement Learning](https://mlsanity.com/paper/2109.08128/) (2021)

## Abstract (arXiv)

We are motivated by the goal of generalist robots that can complete a wide range of tasks across many environments. Critical to this is the robot's ability to acquire some metric of task success or reward, which is necessary for reinforcement learning, planning, or knowing when to ask for help. For a general-purpose robot operating in the real world, this reward function must also be able to generalize broadly across environments, tasks, and objects, while depending only on on-board sensor observations (e.g. RGB images). While deep learning on large and diverse datasets has shown promise as a path towards such generalization in computer vision and natural language, collecting high quality datasets of robotic interaction at scale remains an open challenge. In contrast, "in-the-wild" videos of humans (e.g. YouTube) contain an extensive collection of people doing interesting tasks across a diverse range of settings. In this work, we propose a simple approach, Domain-agnostic Video Discriminator (DVD), that learns multitask reward functions by training a discriminator to classify whether two videos are performing the same task, and can generalize by virtue of learning from a small amount of robot data with a broad dataset of human videos. We find that by leveraging diverse human datasets, this reward function (a) can generalize zero shot to unseen environments, (b) generalize zero shot to unseen tasks, and (c) can be combined with visual model predictive control to solve robotic manipulation tasks on a real WidowX200 robot in an unseen environment from a single human demo.