mlsanityData to 7 Oct 2026

I'M HOI: Inertia-aware Monocular Capture of 3D Human-Object Interactions

arXiv 2312.08869 ↗First posted 10 Dec 2023cs.CVcited by 42

Not on the map. Why?

Abstract (arXiv)

We are living in a world surrounded by diverse and "smart" devices with rich modalities of sensing ability. Conveniently capturing the interactions between us humans and these objects remains far-reaching. In this paper, we present I'm-HOI, a monocular scheme to faithfully capture the 3D motions of both the human and object in a novel setting: using a minimal amount of RGB camera and object-mounted Inertial Measurement Unit (IMU). It combines general motion inference and category-aware refinement. For the former, we introduce a holistic human-object tracking method to fuse the IMU signals and the RGB stream and progressively recover the human motions and subsequently the companion object motions. For the latter, we tailor a category-aware motion diffusion model, which is conditioned on both the raw IMU observations and the results from the previous stage under over-parameterization representation. It significantly refines the initial results and generates vivid body, hand, and object motions. Moreover, we contribute a large dataset with ground truth human and object motions, dense RGB inputs, and rich object-mounted IMU measurements. Extensive experiments demonstrate the effectiveness of I'm-HOI under a hybrid capture setting. Our dataset and code will be released to the community.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 75 of its 111 references that are in the map. First posted on arXiv 10 Dec 2023.

Builds on

75 of 111 references.

References · 75

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2020Denoising Diffusion Implicit Modelscited by 9,655
  4. 2023Segment Anythingcited by 8,158
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  7. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  8. 2022Human Motion Diffusion Modelcited by 1,062
  9. 2017End-to-end Recovery of Human Shape and Posecited by 883
  10. 2020Accelerating 3D Deep Learning with PyTorch3Dcited by 781
  11. 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 692
  12. 2019Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loopcited by 625
38 more
  1. 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 615
  2. 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 497
  3. 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 436
  4. 2022PhysDiff: Physics-Guided Human Motion Diffusion Modelcited by 365
  5. 2023Human Motion Diffusion as a Generative Priorcited by 354
  6. 2021PARE: Part Attention Regressor for 3D Human Body Estimationcited by 334
  7. 2021HuMoR: 3D Human Motion Model for Robust Pose Estimationcited by 319
  8. 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 306
  9. 2020HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimationcited by 304
  10. 2019Resolving 3D Human Pose Ambiguities with 3D Scene Constraintscited by 294
  11. 2019Convolutional Mesh Regression for Single-Image Human Shape Reconstructioncited by 270
  12. 2017Integral Human Pose Regressioncited by 255
  13. 2021PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loopcited by 250
  14. 2022BEHAVE: Dataset and Method for Tracking Human Object Interactionscited by 246
  15. 2018Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodiescited by 239
  16. 2018Learning to Estimate 3D Human Pose and Shape from a Single Color Imagecited by 226
  17. 2023Guided Motion Diffusion for Controllable Human Motion Synthesiscited by 221
  18. 2023Object Motion Guided Human Motion Synthesiscited by 206
  19. 2021Stochastic Scene-Aware Motion Predictioncited by 203
  20. 2020Long-term Human Motion Prediction with Scene Contextcited by 191
  21. 2018Neural Body Fitting: Unifying Deep Learning and Model-Based Human Pose and Shape Estimationcited by 185
  22. 2023InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusioncited by 181
  23. 2017HP-GAN: Probabilistic 3D human motion prediction via GANcited by 176
  24. 2022Capturing and Inferring Dense Full-Body Human-Scene Contactcited by 176
  25. 2022HUMANISE: Language-conditioned Human Motion Generation in 3D Scenescited by 156
  26. 2020Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wildcited by 152
  27. 2021Probabilistic Modeling for Human Mesh Recoverycited by 152
  28. 2022PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Imagescited by 152
  29. 2021GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Camerascited by 151
  30. 2022Physical Inertial Poser (PIP): Physics-aware Real-time Human Motion Tracking from Sparse Inertial Sensorscited by 144
  31. 2022COUCH: Towards Controllable Human-Chair Interactionscited by 134
  32. 2019Generating 3D People in Scenes without Peoplecited by 127
  33. 2022Towards An End-to-End Framework for Flow-Guided Video Inpaintingcited by 111
  34. 2023Synthesizing Diverse Human Motions in 3D Indoor Scenescited by 109
  35. 2017Sparse Inertial Poser: Automatic 3D Human Pose Estimation from Sparse IMUscited by 108
  36. 2021Learning Motion Priors for 4D Human Body Capture in 3D Scenescited by 98
  37. 2022CHORE: Contact, Human and Object REconstruction from a single RGB imagecited by 95
  38. 2022Ego-Body Pose Estimation via Ego-Head Pose Estimationcited by 93

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 16; 2026: 17.

42 papers.

Sort:

By area:

  1. 2025InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactionscited by 67
  2. 2024InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interactioncited by 63
  3. 2024HOI-M3:Capture Multiple Humans and Objects Interaction within Contextual Environmentcited by 52
  4. 2024CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangementcited by 40
  5. 2024GigaHands: A Massive Annotated Dataset of Bimanual Hand Activitiescited by 32
  6. 2025InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generationcited by 31
  7. 2025HUMOTO: A 4D Dataset of Mocap Human Object Interactionscited by 30
  8. 2025A Survey on Human Interaction Motion Generationcited by 25
  9. 2026PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulationcited by 20
  10. 2024TriDi: Trilateral Diffusion of 3D Humans, Objects, and Interactionscited by 18
  11. 2025SemGeoMo: Dynamic Contextual Human Motion Generation with Semantic and Geometric Guidancecited by 16
  12. 2024Mimicking-Bench: A Benchmark for Generalizable Humanoid-Scene Interaction Learning via Human Mimickingcited by 14
30 more
  1. 2025CARI4D: Category Agnostic 4D Reconstruction of Human-Object Interactioncited by 14
  2. 2025Reconstructing In-the-Wild Open-Vocabulary Human-Object Interactionscited by 13
  3. 2025Reconstructing 4D Spatial Intelligence: A Surveycited by 13
  4. 2026InterPrior: Scaling Generative Control for Physics-Based Human-Object Interactionscited by 10
  5. 2026Track4World: Feedforward World-centric Dense 3D Tracking of All Pixelscited by 10
  6. 2025ScoreHOI: Physically Plausible Reconstruction of Human-Object Interaction via Score-Guided Diffusioncited by 9
  7. 2024FORCE: Physics-aware Human-object Interactioncited by 7
  8. 20253D Human Interaction Generation: A Surveycited by 6
  9. 2025ViBES: A Conversational Agent with Behaviorally-Intelligent 3D Virtual Bodycited by 6
  10. 2024Gaze-guided Hand-Object Interaction Synthesis: Dataset and Methodcited by 5
  11. 2025GenM3: Generative Pretrained Multi-path Motion Model for Text Conditional Human Motion Generationcited by 5
  12. 2026Unleashing Guidance Without Classifiers for Human-Object Interaction Animationcited by 5
  13. 2025ECHO: Ego-Centric modeling of Human-Object interactionscited by 4
  14. 2026CoMoVi: Co-Generation of 3D Human Motions and Realistic Videoscited by 4
  15. 2025Efficient and Scalable Monocular Human-Object Interaction Motion Reconstructioncited by 3
  16. 2026AnyLift: Scaling Motion Reconstruction from Internet Videos via 2D Diffusioncited by 2
  17. 20243DArticCyclists: Generating Synthetic Articulated 8D Pose-Controllable Cyclist Data for Computer Vision Applicationscited by 1
  18. 2026End-to-End Spatial-Temporal Transformer for Real-time 4D HOI Reconstructioncited by 1
  19. 2026Uni-HOI:A Unified framework for Learning the Joint distribution of Text and Human-Object Interactioncited by 1
  20. 2026HOIMask: Towards Generative Masked Modeling for Human Object Interaction Generationcited by 1
  21. 2025Physically Plausible Human-Object Rendering from Sparse Views via 3D Gaussian Splattingcited by 0
  22. 2025MoGIC: Boosting Motion Generation via Intention Understanding and Visual Contextcited by 0
  23. 2026LEXIS: LatEnt ProXimal Interaction Signatures for 3D HOI from an Imagecited by 0
  24. 2026Dynamic Full-body Motion Agent with Object Interaction via Blending Pre-trained Modular Controllerscited by 0
  25. 2026Real2Sim in HOI: Toward Physically Plausible HOI Reconstruction from Monocular Videoscited by 0
  26. 2026RHINO: Reconstructing Human Interactions with Novel Objects from Monocular Videoscited by 0
  27. 2026Recovering Physically Plausible Human-Object Interactions from Monocular Videoscited by 0
  28. 2026IMAGIN-4D: Image-Guided Controllable Interaction Generationcited by 0
  29. 2026IMU-HOI: A Symbiotic Framework for Coherent Human-Object Interaction and Motion Capture via Contact-Conscious Inertial Fusioncited by 0
  30. 2026Reconstructing Humans and Objects in Interaction using Large Reconstruction Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026DynamicHOI: Coupled Dynamics for Physics-aware HOI Reconstruction
  2. 2025Human Motion Capture from Loose and Sparse Inertial Sensors with Garment-aware Diffusion Models
  3. 2025Group Inertial Poser: Multi-Person Pose and Global Translation from Sparse Inertial Sensors and Ultra-Wideband Ranging
  4. 2025DiffCap: Diffusion-based Real-time Human Motion Capture using Sparse IMUs and a Monocular Camera
  5. 2025VHOI: Controllable Video Generation of Human-Object Interactions from Sparse Trajectories via Motion Densification
  6. 2025SViMo: Synchronized Diffusion for Video and Motion Generation in Hand-object Interaction Scenarios
  7. 2026HO-Flow: Generalizable Hand-Object Interaction Generation with Latent Flow Matching
  8. 2023DiffusionPoser: Real-time Human Motion Reconstruction From Arbitrary Sparse Sensors Using Autoregressive Diffusion