mlsanityData to 13 Oct 2026

Learning to Imitate Object Interactions from Internet Videos

arXiv 2211.13225 ↗First posted 23 Nov 2022cs.CV cs.LG cs.ROcited by 32

Not on the map. Why?

Abstract (arXiv)

We study the problem of imitating object interactions from Internet videos. This requires understanding the hand-object interactions in 4D, spatially in 3D and over time, which is challenging due to mutual hand-object occlusions. In this paper we make two main contributions: (1) a novel reconstruction technique RHOV (Reconstructing Hands and Objects from Videos), which reconstructs 4D trajectories of both the hand and the object using 2D image cues and temporal smoothness constraints; (2) a system for imitating object interactions in a physics simulator with reinforcement learning. We apply our reconstruction technique to 100 challenging Internet videos. We further show that we can successfully imitate a range of different object interactions in a physics simulator. Our object-centric approach is not limited to human-like end-effectors and can learn to imitate object interactions using different embodiments, like a robotic arm with a parallel jaw gripper.

Key facts. Cited by 32 papers in the map (data through 13 Oct 2026). Builds on 51 of its 66 references that are in the map. First posted on arXiv 23 Nov 2022.

Builds on

51 of 66 references.

References · 51

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  2. 2017Proximal Policy Optimization Algorithmscited by 14,592
  3. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,395
  4. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,535
  5. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,490
  6. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 890
  7. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 792
  8. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 770
  9. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 455
  10. 2019PointRend: Image Segmentation as Renderingcited by 439
  11. 2017Neural 3D Mesh Renderercited by 437
  12. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 398
38 more
  1. 2019Learning joint reconstruction of hands and manipulated objectscited by 377
  2. 2021DexYCB: A Benchmark for Capturing Hand Grasping of Objectscited by 322
  3. 2020CosyPose: Consistent multi-view multi-object 6D pose estimationcited by 287
  4. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
  5. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  6. 2022Masked Visual Pre-training for Motor Controlcited by 246
  7. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 242
  8. 2017Learning to Estimate 3D Hand Pose from Single RGB Imagescited by 234
  9. 2018Pix3D: Dataset and Methods for Single-Image 3D Shape Modelingcited by 218
  10. 2018RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitationcited by 202
  11. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videoscited by 195
  12. 2022Human-to-Robot Imitation in the Wildcited by 158
  13. 2020Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wildcited by 152
  14. 2017GANerated Hands for Real-time 3D Hand Tracking from Monocular RGBcited by 147
  15. 2018One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learningcited by 146
  16. 2020Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wildcited by 146
  17. 2020Reconstructing Hand-Object Interactions in the Wildcited by 143
  18. 2020Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstructioncited by 139
  19. 2015Capturing Hands in Action using Discriminative Salient Points and Physics Simulationcited by 115
  20. 2021D-Grasp: Physically Plausible Dynamic Grasp Synthesis for Hand-Object Interactionscited by 110
  21. 2019DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm Systemcited by 106
  22. 2017V2V-PoseNet: Voxel-to-Voxel Prediction Network for Accurate 3D Hand and Human Pose Estimation from a Single Depth Mapcited by 102
  23. 2020Weakly Supervised 3D Hand Pose Estimation via Biomechanical Constraintscited by 97
  24. 2020State-Only Imitation Learning for Dexterous Manipulationcited by 90
  25. 2022What's in your hands? 3D Reconstruction of Generic Objects in Handscited by 90
  26. 2022Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtubecited by 79
  27. 2022From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperationcited by 79
  28. 2022Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulationcited by 75
  29. 2022QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatarscited by 74
  30. 2020FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integrationcited by 72
  31. 2020Mask2CAD: 3D Shape Prediction by Learning to Segment and Retrievecited by 65
  32. 2021Towards unconstrained joint hand-object reconstruction from RGB videoscited by 57
  33. 2019A2J: Anchor-to-Joint Regression Network for 3D Articulated Pose Estimation from a Single Depth Imagecited by 56
  34. 2022Holo-Dex: Teaching Dexterity with Immersive Mixed Realitycited by 52
  35. 2017Factoring Shape, Pose, and Layout from the 2D Image of a 3D Scenecited by 50
  36. 2018Disentangling Latent Hands for Image Synthesis and Pose Estimationcited by 44
  37. 2017Depth-Based 3D Hand Pose Estimation: From Current Achievements to Future Goalscited by 21
  38. 2020Learning Object Manipulation Skills via Approximate State Estimation from Real Videoscited by 12

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 9; 2024: 7; 2025: 12; 2026: 4.

32 papers.

Sort:

By area:

  1. 2023Humans in 4D: Reconstructing and Tracking Humans with Transformerscited by 345
  2. 2023Rotating without Seeing: Towards In-hand Dexterity through Touchcited by 78
  3. 2023Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction Clipscited by 73
  4. 2023PhysHOI: Physics-Based Imitation of Dynamic Human-Object Interactioncited by 73
  5. 2025Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videoscited by 67
  6. 2026SAM 3D Body: Robust Full-Body Human Mesh Recoverycited by 65
  7. 2023Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensingcited by 52
  8. 2024CityWalker: Learning Embodied Urban Navigation from Web-Scale Videoscited by 41
  9. 2024Hand-Object Interaction Pretraining from Videoscited by 38
  10. 2025Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrationscited by 34
  11. 2025X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Realcited by 25
  12. 2024Reconstructing Hand-Held Objects in 3D from Images and Videoscited by 24
20 more
  1. 2023Look Ma, No Hands! Agent-Environment Factorization of Egocentric Videoscited by 21
  2. 2025Predicting 4D Hand Trajectory from Monocular Videoscited by 19
  3. 2024Bi-KVIL: Keypoints-based Visual Imitation Learning of Bimanual Manipulation Taskscited by 16
  4. 2025Object-centric 3D Motion Field for Robot Learning from Human Videoscited by 13
  5. 2026Dex4D: Task-Agnostic Point Track Policy for Zero-Shot Dexterous Manipulationcited by 12
  6. 2023Get a Grip: Reconstructing Hand-Object Stable Grasps in Egocentric Videoscited by 10
  7. 2024VIEW: Visual Imitation Learning with Waypointscited by 10
  8. 20256D Object Pose Tracking in Internet Videos for Robotic Manipulationcited by 10
  9. 2023Towards Generalist Robots: A Promising Paradigm via Generative Simulationcited by 9
  10. 2025Follow My Hold: Hand-Object Interaction Reconstruction through Geometric Guidancecited by 8
  11. 2025TOUCH: Text-guided Controllable Generation of Free-Form Hand-Object Interactionscited by 8
  12. 2024On-Robot Reinforcement Learning with Goal-Contrastive Rewardscited by 7
  13. 2025DexMan: Learning Bimanual Dexterous Manipulation from Human and Generated Videoscited by 7
  14. 2025Large-scale Pre-training for Grounded Video Caption Generationcited by 5
  15. 2025Reconstructing Objects along Hand Interaction Timelines in Egocentric Videocited by 3
  16. 2025Slot-Level Robotic Placement via Visual Imitation from Single Human Videocited by 2
  17. 2023H-InDex: Visual Reinforcement Learning with Hand-Informed Representations for Dexterous Manipulationcited by 1
  18. 2026D-REX: Differentiable Real-to-Sim-to-Real Engine for Learning Dexterous Graspingcited by 1
  19. 2026Towards in-the-wild Egocentric 3D Hand-Object Pose Estimationcited by 1
  20. 2024Grounded Video Caption Generationcited by 0