mlsanityData to 7 Oct 2026

HOI-M3:Capture Multiple Humans and Objects Interaction within Contextual Environment

arXiv 2404.00299 ↗First posted 30 Mar 2024cs.CVcited by 52

Not on the map. Why?

Abstract (arXiv)

Humans naturally interact with both others and the surrounding multiple objects, engaging in various social activities. However, recent advances in modeling human-object interactions mostly focus on perceiving isolated individuals and objects, due to fundamental data scarcity. In this paper, we introduce HOI-M3, a novel large-scale dataset for modeling the interactions of Multiple huMans and Multiple objects. Notably, it provides accurate 3D tracking for both humans and objects from dense RGB and object-mounted IMU inputs, covering 199 sequences and 181M frames of diverse humans and objects under rich activities. With the unique HOI-M3 dataset, we introduce two novel data-driven tasks with companion strong baselines: monocular capture and unstructured generation of multiple human-object interactions. Extensive experiments demonstrate that our dataset is challenging and worthy of further research about multiple human-object interactions and behavior analysis. Our HOI-M3 dataset, corresponding codes, and pre-trained models will be disseminated to the community for future research.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Builds on 64 of its 84 references that are in the map. First posted on arXiv 30 Mar 2024.

Builds on

64 of 84 references.

References · 64

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2023Segment Anythingcited by 8,158
  4. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  5. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  6. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  7. 2022Human Motion Diffusion Modelcited by 1,062
  8. 2017End-to-end Recovery of Human Shape and Posecited by 883
  9. 2019Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loopcited by 625
  10. 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 615
  11. 2022ViTPose: Simple Vision Transformer Baselines for Human Pose Estimationcited by 450
  12. 2017Neural 3D Mesh Renderercited by 437
38 more
  1. 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 436
  2. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 397
  3. 2021PARE: Part Attention Regressor for 3D Human Body Estimationcited by 334
  4. 2020HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimationcited by 304
  5. 2019Resolving 3D Human Pose Ambiguities with 3D Scene Constraintscited by 294
  6. 2022BEHAVE: Dataset and Method for Tracking Human Object Interactionscited by 246
  7. 2022CLIFF: Carrying Location Information in Full Frames into Human Pose and Shape Estimationcited by 237
  8. 2022MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesiscited by 218
  9. 2020Monocular, One-stage, Regression of Multiple 3D Peoplecited by 216
  10. 2023Object Motion Guided Human Motion Synthesiscited by 206
  11. 2023InterGen: Diffusion-based Multi-human Motion Generation under Complex Interactionscited by 205
  12. 2021Stochastic Scene-Aware Motion Predictioncited by 203
  13. 2017Single-Shot Multi-Person 3D Pose Estimation From Monocular RGBcited by 195
  14. 2020Long-term Human Motion Prediction with Scene Contextcited by 191
  15. 2022Capturing and Inferring Dense Full-Body Human-Scene Contactcited by 176
  16. 2021FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integrationcited by 168
  17. 2021Hand-Object Contact Consistency Reasoning for Human Grasps Generationcited by 160
  18. 2022HUMANISE: Language-conditioned Human Motion Generation in 3D Scenescited by 156
  19. 2020Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wildcited by 152
  20. 2023CG-HOI: Contact-Guided 3D Human-Object Interaction Generationcited by 137
  21. 2024Scaling Up Dynamic Human-Scene Interaction Modelingcited by 137
  22. 2019Camera Distance-aware Top-down Approach for 3D Multi-person Pose Estimation from a Single RGB Imagecited by 131
  23. 2019Generating 3D People in Scenes without Peoplecited by 127
  24. 2015Capturing Hands in Action using Discriminative Salient Points and Physics Simulationcited by 115
  25. 2020We are More than Our Joints: Predicting how 3D Bodies Movecited by 115
  26. 2023Controllable Human-Object Interaction Synthesiscited by 107
  27. 2022CHORE: Contact, Human and Object REconstruction from a single RGB imagecited by 95
  28. 2022Ego-Body Pose Estimation via Ego-Head Pose Estimationcited by 93
  29. 2022InterCap: Joint Markerless 3D Tracking of Humans and Objects in Interactioncited by 90
  30. 2022Balanced MSE for Imbalanced Visual Regressioncited by 89
  31. 2023HOI-Diff: Text-Driven Synthesis of 3D Human-Object Interactions using Diffusion Modelscited by 89
  32. 2023CIRCLE: Capture In Rich Contextual Environmentscited by 79
  33. 2023PhysHOI: Physics-Based Imitation of Dynamic Human-Object Interactioncited by 72
  34. 2022NeuralDome: A Neural Modeling Pipeline on Multi-View Human-Object Interactionscited by 67
  35. 2023Visibility Aware Human-Object Interaction Tracking from Single RGB Cameracited by 58
  36. 2022MIME: Human-Aware 3D Scene Generationcited by 57
  37. 2022Human-Aware Object Placement for Visual Environment Reconstructioncited by 56
  38. 2020Holistic 3D Human and Scene Mesh Estimation from Single View Imagescited by 54

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 23; 2026: 23.

52 papers.

Sort:

By area:

  1. 2025InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactionscited by 67
  2. 2024The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motioncited by 46
  3. 2024CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangementcited by 40
  4. 2025InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generationcited by 31
  5. 2025HUMOTO: A 4D Dataset of Mocap Human Object Interactionscited by 30
  6. 2025A Survey on Human Interaction Motion Generationcited by 25
  7. 2024InterTrack: Tracking Human Object Interaction without Object Templatescited by 22
  8. 2025The Quest for Generalizable Motion Generation: Data, Model, and Evaluationcited by 20
  9. 2026PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulationcited by 20
  10. 2024TriDi: Trilateral Diffusion of 3D Humans, Objects, and Interactionscited by 18
  11. 2025SocialGen: Modeling Multi-Human Social Interaction with Language Modelscited by 14
  12. 2025CARI4D: Category Agnostic 4D Reconstruction of Human-Object Interactioncited by 14
38 more
  1. 2024SCENIC: Scene-aware Semantic Navigation with Instruction-guided Controlcited by 13
  2. 2026InterPrior: Scaling Generative Control for Physics-Based Human-Object Interactionscited by 10
  3. 2025MVHumanNet++: A Large-scale Dataset of Multi-view Daily Dressing Human Captures with Richer Annotations for 3D Human Digitizationcited by 9
  4. 2025Perceiving and Acting in First-Person: A Dataset and Benchmark for Egocentric Human-Object-Human Interactionscited by 7
  5. 2026FrankenMotion: Part-level Human Motion Generation and Compositioncited by 7
  6. 20253D Human Interaction Generation: A Surveycited by 6
  7. 2025ViBES: A Conversational Agent with Behaviorally-Intelligent 3D Virtual Bodycited by 6
  8. 2026Unleashing Guidance Without Classifiers for Human-Object Interaction Animationcited by 5
  9. 2025ECHO: Ego-Centric modeling of Human-Object interactionscited by 4
  10. 2025InterAgent: Physics-based Multi-agent Command Execution via Diffusion on Interaction Graphscited by 4
  11. 2026TeamHOI: Learning a Unified Policy for Cooperative Human-Object Interactions with Any Team Sizecited by 4
  12. 2026SCRIPT: Scalable Diffusion Policy with Multi-stage Training for Language-driven Physics-Based Humanoid Controlcited by 4
  13. 2025Versatile Physics-based Character Control with Hybrid Latent Representationcited by 3
  14. 2025PA-HOI: A Physics-Aware Human and Object Interaction Datasetcited by 3
  15. 2025Efficient and Scalable Monocular Human-Object Interaction Motion Reconstructioncited by 3
  16. 2026PolySLGen: Online Multimodal Speaking-Listening Reaction Generation in Polyadic Interactioncited by 3
  17. 2024COLLAGE: Collaborative Human-Agent Interaction Generation using Hierarchical Latent Diffusion and Language Modelscited by 2
  18. 2025HUMOF: Human Motion Forecasting in Interactive Social Scenescited by 2
  19. 2025MMHOI: Modeling Complex 3D Multi-Human Multi-Object Interactionscited by 2
  20. 2025OmniMotion-X: Versatile Multimodal Whole-Body Motion Generationcited by 2
  21. 2026Not All Frames Are Equal: Complexity-Aware Masked Motion Generation via Motion Spectral Descriptorscited by 2
  22. 2026End-to-End Spatial-Temporal Transformer for Real-time 4D HOI Reconstructioncited by 1
  23. 2026MOCHI: Motion Enhancement of Collaborative Human-object Interactionscited by 1
  24. 2026SSMNBench: Diagnosing Image-based Cross-View Human-Object Understanding via Single-View Sufficiency and Multi-View Necessitycited by 1
  25. 2026GIRAF: Towards Generalizable Human Interactions with Articulated Objectscited by 1
  26. 2025Physically Plausible Human-Object Rendering from Sparse Views via 3D Gaussian Splattingcited by 0
  27. 2025MoGIC: Boosting Motion Generation via Intention Understanding and Visual Contextcited by 0
  28. 2025Learning to Generate Human-Human-Object Interactions from Textual Descriptionscited by 0
  29. 2025Asset-Driven Sematic Reconstruction of Dynamic Scene with Multi-Human-Object Interactionscited by 0
  30. 2026Rendering Multi-Human and Multi-Object with 3D Gaussian Splattingcited by 0
  31. 2026Action Motifs: Self-Supervised Hierarchical Representation of Human Body Movementscited by 0
  32. 2026OmniRobotHome: A Multi-Camera Home Platform for Real-Time Human-Robot Interactioncited by 0
  33. 2026Real2Sim in HOI: Toward Physically Plausible HOI Reconstruction from Monocular Videoscited by 0
  34. 2026RHINO: Reconstructing Human Interactions with Novel Objects from Monocular Videoscited by 0
  35. 2026IMAGIN-4D: Image-Guided Controllable Interaction Generationcited by 0
  36. 2026IMU-HOI: A Symbiotic Framework for Coherent Human-Object Interaction and Motion Capture via Contact-Conscious Inertial Fusioncited by 0
  37. 2026RF-HOI: Recognize Human-Object Interaction with Radio Frequency Signalscited by 0
  38. 2026Super Star: Towards Streaming Real-time Interactive Agents for Digital Humanscited by 0

50 most cited of 52 All 52 (JSON)