mlsanityData to 7 Oct 2026

HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction

arXiv 2203.01577 ↗First posted 3 Mar 2022cs.CVcited by 321

Not on the map. Why?

Abstract (arXiv)

We present HOI4D, a large-scale 4D egocentric dataset with rich annotations, to catalyze the research of category-level human-object interaction. HOI4D consists of 2.4M RGB-D egocentric video frames over 4000 sequences collected by 4 participants interacting with 800 different object instances from 16 categories over 610 different indoor rooms. Frame-wise annotations for panoptic segmentation, motion segmentation, 3D hand pose, category-level object pose and hand action have also been provided, together with reconstructed object meshes and scene point clouds. With HOI4D, we establish three benchmarking tasks to promote category-level HOI from 4D visual signals including semantic segmentation of 4D dynamic point cloud sequences, category-level object pose tracking, and egocentric action segmentation with diverse interaction targets. In-depth analysis shows HOI4D poses great challenges to existing methods and produces great research opportunities.

Key facts. Cited by 321 papers in the map (data through 7 Oct 2026). Builds on 36 of its 54 references that are in the map. First posted on arXiv 3 Mar 2022.

Builds on

36 of 54 references.

References · 36

  1. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  2. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  3. 2016Generative Adversarial Imitation Learningcited by 1,576
  4. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
  5. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  6. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  7. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 768
  8. 2018Open3D: A Modern Library for 3D Data Processingcited by 717
  9. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 584
  10. 2018PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understandingcited by 521
  11. 2019Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimationcited by 490
  12. 2019Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoningcited by 420
24 more
  1. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 397
  2. 2019Learning joint reconstruction of hands and manipulated objectscited by 377
  3. 2019MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentationcited by 370
  4. 2021DexYCB: A Benchmark for Capturing Hand Grasping of Objectscited by 320
  5. 2019FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Imagescited by 271
  6. 2021Mesh Graphormercited by 264
  7. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 251
  8. 2017Hand Keypoint Detection in Single Images using Multiview Bootstrappingcited by 231
  9. 2017First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotationscited by 229
  10. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videoscited by 194
  11. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 193
  12. 2020ContactPose: A Dataset of Grasps with Object Contact and Hand Posecited by 180
  13. 2021ASFormer: Transformer for Action Segmentationcited by 166
  14. 2019Category-Level Articulated Object Pose Estimationcited by 158
  15. 2021Modular Interactive Video Object Segmentation: Interaction-to-Mask, Propagation and Difference-Aware Fusioncited by 129
  16. 2019MeteorNet: Deep Learning on Dynamic 3D Point Cloud Sequencescited by 117
  17. 2019DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm Systemcited by 105
  18. 2021BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Modelscited by 87
  19. 2021CAPTRA: CAtegory-level Pose Tracking for Rigid and Articulated Objects from Point Cloudscited by 83
  20. 20196-PACK: Category-level 6D Pose Tracker with Anchor-Based Keypointscited by 78
  21. 20214D Panoptic LiDAR Segmentationcited by 64
  22. 2017BigHand2.2M Benchmark: Hand Pose Dataset and State of the Art Analysiscited by 46
  23. 20214D-Net for Learned Multi-Modal Alignmentcited by 44
  24. 2016Model-based Deep Hand Pose Estimationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 39; 2024: 60; 2025: 99; 2026: 118.

321 papers.

Sort:

By area:

  1. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  2. 20243D-VLA: A 3D Vision-Language-Action Generative World Modelcited by 320
  3. 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 306
  4. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  5. 2025EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Videocited by 191
  6. 2025UniDepthV2: Universal Monocular Metric Depth Estimation Made Simplercited by 170
  7. 2023Aria Digital Twin: A New Benchmark Dataset for Egocentric 3D Machine Perceptioncited by 166
  8. 2023HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real Worldcited by 159
  9. 2024HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videoscited by 122
  10. 2022Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?cited by 121
  11. 2023UniDexGrasp: Universal Robotic Dexterous Grasping via Learning Diverse Proposal Generation and Goal-Conditioned Policycited by 117
  12. 2025EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videoscited by 116
38 more
  1. 2023Affordance Diffusion: Synthesizing Hand-Object Interactionscited by 110
  2. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Modelscited by 105
  3. 2023UniDexGrasp++: Improving Dexterous Grasping Policy Learning via Geometry-aware Curriculum and Iterative Generalist-Specialist Learningcited by 101
  4. 2025HD-EPIC: A Highly-Detailed Egocentric Video Datasetcited by 101
  5. 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videoscited by 94
  6. 2024EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real Worldcited by 93
  7. 2025EgoLife: Towards Egocentric Life Assistantcited by 86
  8. 2024TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understandingcited by 85
  9. 2023An Outlook into the Future of Egocentric Visioncited by 84
  10. 2025SpatialTrackerV2: 3D Point Tracking Made Easycited by 84
  11. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  12. 2024General Flow as Foundation Affordance for Scalable Robot Learningcited by 80
  13. 2024OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completioncited by 78
  14. 2025STream3R: Scalable Sequential 3D Reconstruction with Causal Transformercited by 76
  15. 2023Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction Clipscited by 73
  16. 2024TAPVid-3D: A Benchmark for Tracking Any Point in 3Dcited by 69
  17. 2026Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalizationcited by 61
  18. 2025UniK3D: Universal Camera Monocular 3D Estimationcited by 60
  19. 2025OmniWorld: A Multi-Domain and Multi-Modal Dataset for 4D World Modelingcited by 59
  20. 2025villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Modelscited by 58
  21. 2025ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learningcited by 56
  22. 2024RAM: Retrieval-Based Affordance Transfer for Generalizable Zero-Shot Robotic Manipulationcited by 55
  23. 2023CAMS: CAnonicalized Manipulation Spaces for Category-Level Functional Hand-Object Manipulation Synthesiscited by 54
  24. 2024ParaHome: Parameterizing Everyday Home Activities Towards 3D Generative Modeling of Human-Object Interactionscited by 54
  25. 2025OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature Extractioncited by 54
  26. 2023ArtiGrasp: Physically Plausible Synthesis of Bi-Manual Dexterous Grasping and Articulationcited by 53
  27. 2024Dyn-HaMR: Recovering 4D Interacting Hand Motion from a Dynamic Cameracited by 50
  28. 2026EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the Worldcited by 49
  29. 2024Introducing HOT3D: An Egocentric Dataset for 3D Hand and Object Trackingcited by 46
  30. 2025SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulationcited by 46
  31. 2025VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulationcited by 46
  32. 2026World Action Models: The Next Frontier in Embodied AIcited by 46
  33. 2024G-HOP: Generative Hand-Object Prior for Interaction Reconstruction and Grasp Synthesiscited by 43
  34. 2025H-RDT: Human Manipulation Enhanced Bimanual Robotic Manipulationcited by 41
  35. 2024CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangementcited by 40
  36. 2024HRP: Human Affordances for Robotic Pre-Trainingcited by 40
  37. 2023EgoObjects: A Large-Scale Egocentric Dataset for Fine-Grained Object Understandingcited by 39
  38. 2024Deep Learning-Based Object Pose Estimation: A Comprehensive Surveycited by 39

50 most cited of 321 All 321 (JSON)