mlsanityData to 7 Oct 2026

AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos

arXiv 1912.04443 ↗First posted 10 Dec 2019cs.RO cs.CV cs.LGcited by 105

Not on the map. Why?

Cited by 28 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Robotic reinforcement learning (RL) holds the promise of enabling robots to learn complex behaviors through experience. However, realizing this promise for long-horizon tasks in the real world requires mechanisms to reduce human burden in terms of defining the task and scaffolding the learning process. In this paper, we study how these challenges can be alleviated with an automated robotic learning framework, in which multi-stage tasks are defined simply by providing videos of a human demonstrator and then learned autonomously by the robot from raw image observations. A central challenge in imitating human videos is the difference in appearance between the human and robot, which typically requires manual correspondence. We instead take an automated approach and perform pixel-level image translation via CycleGAN to convert the human demonstration into a video of a robot, which can then be used to construct a reward function for a model-based RL algorithm. The robot then learns the task one stage at a time, automatically learning how to reset each stage to retry it multiple times without human-provided resets. This makes the learning process largely automatic, from intuitive task specification via a video to automated training with minimal human intervention. We demonstrate that our approach is capable of learning complex tasks, such as operating a coffee machine, directly from raw image observations, requiring only 20 minutes to provide human demonstrations and about 180 minutes of robot interaction.

Key facts. Cited by 105 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 2; 2021: 12; 2022: 19; 2023: 20; 2024: 29; 2025: 15; 2026: 8.

105 papers.

Sort:

By area:

  1. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  2. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  3. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  4. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 377
  5. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  6. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  7. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 214
  8. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videoscited by 194
  9. 2024HumanPlus: Humanoid Shadowing and Imitation from Humanscited by 164
  10. 2023Language-Driven Representation Learning for Roboticscited by 163
  11. 2022Human-to-Robot Imitation in the Wildcited by 157
  12. 2024DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulationcited by 149
38 more
  1. 2023Structured World Models from Human Videoscited by 144
  2. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 142
  3. 2024Flow as the Cross-Domain Manipulation Interfacecited by 126
  4. 2021Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videoscited by 116
  5. 2024Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulationcited by 116
  6. 2021XIRL: Cross-embodiment Inverse Reinforcement Learningcited by 101
  7. 2022VideoDex: Learning Dexterity from Internet Videoscited by 101
  8. 2021Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Interventioncited by 90
  9. 2021Learning by Watching: Physical Imitation of Manipulation Skills from Human Videoscited by 87
  10. 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 85
  11. 2021Bottom-Up Skill Discovery from Unsegmented Demonstrations for Long-Horizon Robot Manipulationcited by 82
  12. 2023XSkill: Cross Embodiment Skill Discoverycited by 74
  13. 2023Advances and Challenges in Meta-Learning: A Technical Reviewcited by 65
  14. 2025Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learningcited by 57
  15. 2023Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Planscited by 51
  16. 2023Zero-Shot Robot Manipulation from Passive Human Videoscited by 49
  17. 2026EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the Worldcited by 49
  18. 2024Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformerscited by 47
  19. 2023Towards A Unified Agent with Foundation Modelscited by 46
  20. 2024Vision-based Manipulation from Single Human Video with Open-World Object Graphscited by 46
  21. 2025VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulationcited by 46
  22. 2021Cross-domain Imitation from Observationscited by 42
  23. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  24. 2024Surgical Robot Transformer (SRT): Imitation Learning for Surgical Taskscited by 38
  25. 2024Hand-Object Interaction Pretraining from Videoscited by 38
  26. 2023Video-Mined Task Graphs for Keystep Recognition in Instructional Videoscited by 35
  27. 2024ViViDex: Learning Vision-based Dexterous Manipulation from Human Videoscited by 33
  28. 2025Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrationscited by 33
  29. 2022HAKE: A Knowledge Engine Foundation for Human Activity Understandingcited by 28
  30. 2022K-VIL: Keypoints-based Visual Imitation Learningcited by 27
  31. 2025EgoBridge: Domain Adaptation for Generalizable Imitation from Egocentric Human Datacited by 27
  32. 2024Bridging the Human to Robot Dexterity Gap through Object-Oriented Rewardscited by 25
  33. 2022Goal-conditioned dual-action imitation learning for dexterous dual-arm robot manipulationcited by 21
  34. 2023Look Ma, No Hands! Agent-Environment Factorization of Egocentric Videoscited by 21
  35. 2021Domain-Robust Visual Imitation Learning with Mutual Information Constraintscited by 20
  36. 2024Continuously Improving Mobile Manipulation with Autonomous Real-World RLcited by 20
  37. 2023From Isolated Islands to Pangea: Unifying Semantic Space for Human Action Understandingcited by 19
  38. 2025Robot Learning from a Physical World Modelcited by 18

50 most cited of 105 All 105 (JSON)