mlsanityData to 7 Oct 2026

Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videos

arXiv 2103.16817 ↗First posted 31 Mar 2021cs.RO cs.AI cs.CVcited by 116

Not on the map. Why?

Cited by 20 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We are motivated by the goal of generalist robots that can complete a wide range of tasks across many environments. Critical to this is the robot's ability to acquire some metric of task success or reward, which is necessary for reinforcement learning, planning, or knowing when to ask for help. For a general-purpose robot operating in the real world, this reward function must also be able to generalize broadly across environments, tasks, and objects, while depending only on on-board sensor observations (e.g. RGB images). While deep learning on large and diverse datasets has shown promise as a path towards such generalization in computer vision and natural language, collecting high quality datasets of robotic interaction at scale remains an open challenge. In contrast, "in-the-wild" videos of humans (e.g. YouTube) contain an extensive collection of people doing interesting tasks across a diverse range of settings. In this work, we propose a simple approach, Domain-agnostic Video Discriminator (DVD), that learns multitask reward functions by training a discriminator to classify whether two videos are performing the same task, and can generalize by virtue of learning from a small amount of robot data with a broad dataset of human videos. We find that by leveraging diverse human datasets, this reward function (a) can generalize zero shot to unseen environments, (b) generalize zero shot to unseen tasks, and (c) can be combined with visual model predictive control to solve robotic manipulation tasks on a real WidowX200 robot in an unseen environment from a single human demo.

Key facts. Cited by 116 papers in the map (data through 7 Oct 2026). Builds on 43 of its 61 references that are in the map. First posted on arXiv 31 Mar 2021.

Builds on

43 of 61 references.

References · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  4. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  5. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  6. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  7. 2018Learning Dexterous In-Hand Manipulationcited by 643
  8. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  9. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  10. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  11. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
  12. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 266
31 more
  1. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  2. 2017Stochastic Variational Video Predictioncited by 234
  3. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 220
  4. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  5. 2018RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitationcited by 202
  6. 2015Maximum Entropy Deep Inverse Reinforcement Learningcited by 173
  7. 2017Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translationcited by 167
  8. 2018One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learningcited by 146
  9. 2018Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learningcited by 121
  10. 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 115
  11. 2020The Ingredients of Real-World Robotic Reinforcement Learningcited by 111
  12. 2020Visual Imitation Made Easycited by 111
  13. 2019AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videoscited by 105
  14. 2021Learning by Watching: Physical Imitation of Manipulation Skills from Human Videoscited by 87
  15. 2020Reinforcement Learning with Videos: Combining Offline Observations with Interactioncited by 80
  16. 2018Robot Learning in Homes: Improving Generalization and Reducing Dataset Biascited by 75
  17. 2019Third-Person Visual Imitation Learning via Decoupled Hierarchical Controllercited by 73
  18. 2018Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definitioncited by 66
  19. 2016Unsupervised Perceptual Rewards for Imitation Learningcited by 64
  20. 2020Semantic Visual Navigation by Watching YouTube Videoscited by 60
  21. 2020Model-Based Inverse Reinforcement Learning from Visual Demonstrationscited by 52
  22. 2019Learning Predictive Models From Observation and Interactioncited by 41
  23. 2019Learning One-Shot Imitation from Humans without Humanscited by 32
  24. 2017Translating Videos to Commands for Robotic Manipulation with Deep Recurrent Neural Networkscited by 21
  25. 2020Scalable Multi-Task Imitation Learning with Autonomous Improvementcited by 21
  26. 2018One-Shot Learning of Multi-Step Tasks from Observation via Activity Localization in Auxiliary Videocited by 16
  27. 2020Batch Exploration with Examples for Scalable Robotic Reinforcement Learningcited by 14
  28. 2019Online Object Representations with Contrastive Learningcited by 13
  29. 2020Learning Object Manipulation Skills via Approximate State Estimation from Real Videoscited by 12
  30. 2018Deep Episodic Memory: Encoding, Recalling, and Predicting Episodic Experiences for Robot Action Executioncited by 11
  31. 2019Perceptual Values from Observationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 17; 2023: 26; 2024: 35; 2025: 22; 2026: 11.

116 papers.

Sort:

By area:

  1. 2022A Generalist Agentcited by 802
  2. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  3. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  4. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  5. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 377
  6. 2025π*0.6: a VLA That Learns From Experiencecited by 280
  7. 2023Reward Design with Language Modelscited by 257
  8. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  9. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 214
  10. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
  11. 2024HumanPlus: Humanoid Shadowing and Imitation from Humanscited by 164
  12. 2023Language-Driven Representation Learning for Roboticscited by 163
38 more
  1. 2023Learning to Act from Actionless Videos through Dense Correspondencescited by 159
  2. 2022Human-to-Robot Imitation in the Wildcited by 157
  3. 2023Structured World Models from Human Videoscited by 144
  4. 2024Data Scaling Laws in Imitation Learning for Robotic Manipulationcited by 144
  5. 2023Vision-Language Models as Success Detectorscited by 138
  6. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 128
  7. 2024Flow as the Cross-Domain Manipulation Interfacecited by 126
  8. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  9. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  10. 2022DALL-E-Bot: Introducing Web-Scale Diffusion Models to Roboticscited by 105
  11. 2022VideoDex: Learning Dexterity from Internet Videoscited by 101
  12. 2022Deep Hierarchical Planning from Pixelscited by 96
  13. 2022Instruction-driven history-aware policies for robotic manipulationscited by 93
  14. 2024Dreamitate: Real-World Visuomotor Policy Learning via Video Generationcited by 84
  15. 2024Vision Language Models are In-Context Value Learnerscited by 84
  16. 2023XSkill: Cross Embodiment Skill Discoverycited by 74
  17. 2025Humanoid Policy ~ Human Policycited by 65
  18. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learningcited by 64
  19. 2023Diffusion Reward: Learning Rewards via Conditional Video Diffusioncited by 56
  20. 2022Retrospectives on the Embodied AI Workshopcited by 46
  21. 2024Vision-based Manipulation from Single Human Video with Open-World Object Graphscited by 46
  22. 2025VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulationcited by 46
  23. 2022Graph Inverse Reinforcement Learning from Diverse Videoscited by 45
  24. 2023Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learningcited by 45
  25. 2024DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Controlcited by 43
  26. 2023Language Reward Modulation for Pretraining Reinforcement Learningcited by 42
  27. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  28. 2024CityWalker: Learning Embodied Urban Navigation from Web-Scale Videoscited by 40
  29. 2023GenHowTo: Learning to Generate Actions and State Transformations from Instructional Videoscited by 39
  30. 2024Surgical Robot Transformer (SRT): Imitation Learning for Surgical Taskscited by 38
  31. 2024R+X: Retrieval and Execution from Everyday Human Videoscited by 36
  32. 2025Emergence of Human to Robot Transfer in Vision-Language-Action Modelscited by 36
  33. 2024ViViDex: Learning Vision-based Dexterous Manipulation from Human Videoscited by 33
  34. 2025ZeroMimic: Distilling Robotic Manipulation Skills from Web Videoscited by 31
  35. 2023Polybot: Training One Policy Across Robots While Embracing Variabilitycited by 28
  36. 2024EPO: Hierarchical LLM Agents with Environment Preference Optimizationcited by 28
  37. 2025Self-Improving Embodied Foundation Modelscited by 28
  38. 2024Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Trainingcited by 26

50 most cited of 116 All 116 (JSON)