mlsanityData to 7 Oct 2026

Affordances from Human Videos as a Versatile Representation for Robotics

arXiv 2304.08488 ↗First posted 17 Apr 2023cs.RO cs.AI cs.CVcited by 221

Not on the map. Why?

Cited by 27 in Vision-language-action models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Building a robot that can understand and learn to interact by watching humans has inspired several vision problems. However, despite some successful results on static datasets, it remains unclear how current models can be used on a robot directly. In this paper, we aim to bridge this gap by leveraging videos of human interactions in an environment centric manner. Utilizing internet videos of human behavior, we train a visual affordance model that estimates where and how in the scene a human is likely to interact. The structure of these behavioral affordances directly enables the robot to perform many complex tasks. We show how to seamlessly integrate our affordance model with four robot learning paradigms including offline imitation learning, exploration, goal-conditioned learning, and action parameterization for reinforcement learning. We show the efficacy of our approach, which we call VRB, across 4 real world environments, over 10 different tasks, and 2 robotic platforms operating in the wild. Results, visualizations and videos at https://robo-affordances.github.io/

Key facts. Cited by 221 papers in the map (data through 7 Oct 2026). Builds on 98 of its 142 references that are in the map. First posted on arXiv 17 Apr 2023.

Builds on

98 of 142 references.

References · 98

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2017Mask R-CNNcited by 7,259
  6. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  7. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  9. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  10. 2017Hindsight Experience Replaycited by 1,096
  11. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  12. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
38 more
  1. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  2. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  3. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  4. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  5. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  6. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  7. 2020Rescaling Egocentric Visioncited by 529
  8. 2014Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecturecited by 483
  9. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  10. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  11. 2016Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learningcited by 351
  12. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  13. 2022HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interactioncited by 321
  14. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  15. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
  16. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  17. 2017Count-Based Exploration with Neural Density Modelscited by 255
  18. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  19. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 248
  20. 2022Masked Visual Pre-training for Motor Controlcited by 246
  21. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  22. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  23. 2021Anticipative Video Transformercited by 197
  24. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videoscited by 194
  25. 2018Differentiable MPC for End-to-end Planning and Controlcited by 178
  26. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 171
  27. 2017Decomposing Motion and Content for Natural Video Sequence Predictioncited by 168
  28. 2021FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integrationcited by 168
  29. 2022Human-to-Robot Imitation in the Wildcited by 157
  30. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  31. 2021The Surprising Effectiveness of Representation Learning for Visual Imitationcited by 149
  32. 2018One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learningcited by 146
  33. 2020Reconstructing Hand-Object Interactions in the Wildcited by 143
  34. 2022EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relationscited by 139
  35. 2018Grounded Human-Object Interaction Hotspots from Videocited by 128
  36. 2018Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learningcited by 121
  37. 2020Rolling-Unrolling LSTMs for Action Anticipation from First-Person Videocited by 119
  38. 2021Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videoscited by 116

50 most cited of 98 All 98 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 20; 2024: 70; 2025: 74; 2026: 57.

221 papers.

Sort:

By area:

  1. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  2. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  3. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 331
  4. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
  5. 2024Latent Action Pretraining from Videoscited by 302
  6. 2025Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasetscited by 218
  7. 2023Any-point Trajectory Modeling for Policy Learningcited by 207
  8. 2025EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Videocited by 191
  9. 2025DreamGen: Unlocking Generalization in Robot Learning through Video World Modelscited by 168
  10. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 167
  11. 2024RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulationcited by 166
  12. 2024EgoMimic: Scaling Imitation Learning via Egocentric Videocited by 160
38 more
  1. 2024Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulationcited by 159
  2. 2023Structured World Models from Human Videoscited by 144
  3. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 142
  4. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 136
  5. 2023Large Language Models for Robotics: A Surveycited by 132
  6. 2026π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilitiescited by 127
  7. 2024MOKA: Open-World Robotic Manipulation through Mark-Based Visual Promptingcited by 126
  8. 2024Flow as the Cross-Domain Manipulation Interfacecited by 126
  9. 2025RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twinscited by 117
  10. 2024Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulationcited by 116
  11. 2025EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videoscited by 116
  12. 2025GR-3 Technical Reportcited by 110
  13. 2025FLARE: Robot Learning with Implicit World Modelingcited by 105
  14. 2023On Bringing Robots Homecited by 101
  15. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  16. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  17. 2024General Flow as Foundation Affordance for Scalable Robot Learningcited by 80
  18. 2024HiRT: Enhancing Robotic Control with Hierarchical Robot Transformerscited by 72
  19. 2024Robo-ABC: Affordance Generalization Beyond Categories via Semantic Correspondence for Robot Manipulationcited by 69
  20. 2024RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins (early version)cited by 68
  21. 2025Humanoid Policy ~ Human Policycited by 65
  22. 2025Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videoscited by 63
  23. 2026Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalizationcited by 61
  24. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 60
  25. 2023QUAR-VLA: Vision-Language-Action Model for Quadruped Robotscited by 55
  26. 2024AffordanceLLM: Grounding Affordance from Vision Language Modelscited by 55
  27. 2024RAM: Retrieval-Based Affordance Transfer for Generalizable Zero-Shot Robotic Manipulationcited by 55
  28. 2024Affordance-based Robot Manipulation with Flow Matchingcited by 54
  29. 2023Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Planscited by 51
  30. 2024Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Taskscited by 51
  31. 2026EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the Worldcited by 49
  32. 2024Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformerscited by 47
  33. 2025VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulationcited by 46
  34. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  35. 2023One-Shot Open Affordance Learning with Foundation Modelscited by 45
  36. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  37. 2024HRP: Human Affordances for Robotic Pre-Trainingcited by 40
  38. 2024RT-Affordance: Affordances are Versatile Intermediate Representations for Robot Manipulationcited by 40

50 most cited of 221 All 221 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026AffordanceWAM: Affordance-Aware Joint World-Action Modeling for Robot Manipulation
  2. 2026A4A: Cross-Embodiment Transfer of Action-Oriented 4D Affordances from Human Demonstrations ↗
  3. 2024Learning from 10 Demos: Generalisable and Sample-Efficient Policy Learning with Oriented Affordance Frames
  4. 2026Human-to-Robot Interaction: Learning from Video Demonstration for Robot Imitation
  5. 2026AffordVLA: Injecting Affordance Representations into Vision-Language-Action Models via Implicit Feature Alignment
  6. 2024Affordance-Guided Reinforcement Learning via Visual Prompting
  7. 2026Afford-VLA: Action-Aligned Visual Planning via Internalized Affordance
  8. 2026KAM-WM: Kinematic Affordance Maps from Latent World Models for Robot Manipulation