mlsanityData to 7 Oct 2026

Structured World Models from Human Videos

arXiv 2308.10901 ↗First posted 21 Aug 2023cs.RO cs.AI cs.CVcited by 144

On the map: World models → Embodied and robot world models

Abstract (arXiv)

We tackle the problem of learning complex, general behaviors directly in the real world. We propose an approach for robots to efficiently learn manipulation skills using only a handful of real-world interaction trajectories from many different settings. Inspired by the success of learning from large-scale datasets in the fields of computer vision and natural language, our belief is that in order to efficiently learn, a robot must be able to leverage internet-scale, human video data. Humans interact with the world in many interesting ways, which can allow a robot to not only build an understanding of useful actions and affordances but also how these actions affect the world for manipulation. Our approach builds a structured, human-centric action space grounded in visual affordances learned from human videos. Further, we train a world model on human videos and fine-tune on a small amount of robot interaction data without any task supervision. We show that this approach of affordance-space world models enables different robots to learn various manipulation skills in complex settings, in under 30 minutes of interaction. Videos can be found at https://human-world-model.github.io

Key facts. Cited by 144 papers in the map (data through 7 Oct 2026). Placed in Embodied and robot world models, one of 11 directions in World models. Builds on 67 of its 85 references that are in the map. First posted on arXiv 21 Aug 2023.

Builds on

67 of 85 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning4 Feb
  2. 2021Discovering and Achieving Goals via World Models18 Oct

Other references · 65

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  6. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  7. 2018World Modelscited by 1,444
  8. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  9. 2020Mastering Atari with Discrete World Modelscited by 990
  10. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  11. 2022A Generalist Agentcited by 802
  12. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
38 more
  1. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  2. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  3. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  4. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  5. 2018Learning Dexterous In-Hand Manipulationcited by 643
  6. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 635
  7. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  8. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  9. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  10. 2016The Option-Critic Architecturecited by 433
  11. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 430
  12. 2022DayDreamer: World Models for Physical Robot Learningcited by 372
  13. 2020Planning to Explore via Self-Supervised World Modelscited by 365
  14. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  15. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 288
  16. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  17. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  18. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
  19. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 248
  20. 2022Masked Visual Pre-training for Motor Controlcited by 246
  21. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  22. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
  23. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  24. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  25. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 220
  26. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  27. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 200
  28. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videoscited by 194
  29. 2018Stochastic Adversarial Video Predictioncited by 173
  30. 2022Human-to-Robot Imitation in the Wildcited by 157
  31. 2017Imagination-Augmented Agents for Deep Reinforcement Learningcited by 137
  32. 2018Grounded Human-Object Interaction Hotspots from Videocited by 128
  33. 2018Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learningcited by 121
  34. 2021Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisationcited by 120
  35. 2021Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videoscited by 116
  36. 2022Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videoscited by 110
  37. 2019AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videoscited by 105
  38. 2021XIRL: Cross-embodiment Inverse Reinforcement Learningcited by 101

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 51; 2025: 56; 2026: 30.

144 papers.

Sort:

By area:

  1. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  2. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  3. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 502
  4. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 411
  5. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 386
  6. 2024DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planningcited by 357
  7. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  8. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  9. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  10. 2023Any-point Trajectory Modeling for Policy Learningcited by 207
  11. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 176
  12. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
38 more
  1. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 138
  2. 2024Open-TeleVision: Teleoperation with Immersive Active Visual Feedbackcited by 137
  3. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 136
  4. 2024Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulationcited by 135
  5. 2024Flow as the Cross-Domain Manipulation Interfacecited by 126
  6. 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 117
  7. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  8. 2025GR-3 Technical Reportcited by 110
  9. 2024Scaling Cross-Embodied Learning: One Policy for Manipulation, Navigation, Locomotion and Aviationcited by 108
  10. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 108
  11. 2024ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulationcited by 106
  12. 2024DriveWorld: 4D Pre-trained Scene Understanding via World Models for Autonomous Drivingcited by 89
  13. 2024General Flow as Foundation Affordance for Scalable Robot Learningcited by 80
  14. 2024World Models for Autonomous Driving: An Initial Surveycited by 79
  15. 2024Robo-ABC: Affordance Generalization Beyond Categories via Semantic Correspondence for Robot Manipulationcited by 69
  16. 2024IGOR: Image-GOal Representations are the Atomic Control Units for Foundation Models in Embodied AIcited by 68
  17. 2023EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planningcited by 66
  18. 2025Humanoid Policy ~ Human Policycited by 65
  19. 2025WorldEval: World Model as Real-World Robot Policies Evaluatorcited by 64
  20. 2025Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videoscited by 63
  21. 2025villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Modelscited by 58
  22. 2024A Survey on Robotics with Foundation Models: toward Embodied AIcited by 53
  23. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 50
  24. 2024Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformerscited by 47
  25. 2026World Action Models: The Next Frontier in Embodied AIcited by 46
  26. 2025Generalizing Safety Beyond Collision-Avoidance via Latent-Space Reachability Analysiscited by 44
  27. 2025ReSim: Reliable World Simulation for Autonomous Drivingcited by 42
  28. 2024FLIP: Flow-Centric Generative Planning as General-Purpose Manipulation World Modelcited by 41
  29. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  30. 2024GR-MG: Leveraging Partially Annotated Data via Multi-Modal Goal-Conditioned Policycited by 39
  31. 2024Autoregressive Models in Vision: A Surveycited by 39
  32. 2026RISE: Self-Improving Robot Policy with Compositional World Modelcited by 39
  33. 2024Hand-Object Interaction Pretraining from Videoscited by 38
  34. 2023Model-Based Runtime Monitoring with Interactive Imitation Learningcited by 37
  35. 2025CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learningcited by 37
  36. 2024Learning Manipulation by Predicting Interactioncited by 36
  37. 2024VLM See, Robot Do: Human Demo Video to Robot Action Plan via Vision Language Modelcited by 36
  38. 2024AVID: Adapting Video Diffusion Models to World Modelscited by 35

50 most cited of 144 All 144 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026From Human Videos to Robot Manipulation: A Survey on Scalable Vision-Language-Action Learning with Human-Centric Data
  2. 2026AffordanceWAM: Affordance-Aware Joint World-Action Modeling for Robot Manipulation
  3. 2026Say, Dream, and Act: Learning Video World Models for Instruction-Driven Robot Manipulation
  4. 2026World Models for Robotic Manipulation: A Survey
  5. 20253DFlowAction: Learning Cross-Embodiment Manipulation from 3D Flow World Model
  6. 2026JoyAI-RA 0.5: Scaling Robot Manipulation Learning via Dual Action Alignment
  7. 2026Zero-WAM: In-Context World-Action Modeling from Human Videos for Open-Ended Task Generalization
  8. 2026RoboWM-Bench: A Benchmark for Evaluating World Models in Robotic Manipulation