mlsanityData to 14 Oct 2026

Reconstructing In-the-Wild Open-Vocabulary Human-Object Interactions

arXiv 2503.15898 ↗First posted 20 Mar 2025cs.CVcited by 13

Not on the map. Why?

Abstract (arXiv)

Reconstructing human-object interactions (HOI) from single images is fundamental in computer vision. Existing methods are primarily trained and tested on indoor scenes due to the lack of 3D data, particularly constrained by the object variety, making it challenging to generalize to real-world scenes with a wide range of objects. The limitations of previous 3D HOI datasets were primarily due to the difficulty in acquiring 3D object assets. However, with the development of 3D reconstruction from single images, recently it has become possible to reconstruct various objects from 2D HOI images. We therefore propose a pipeline for annotating fine-grained 3D humans, objects, and their interactions from single images. We annotated 2.5k+ 3D HOI assets from existing 2D HOI datasets and built the first open-vocabulary in-the-wild 3D HOI dataset Open3DHOI, to serve as a future test set. Moreover, we design a novel Gaussian-HOI optimizer, which efficiently reconstructs the spatial interactions between humans and objects while learning the contact regions. Besides the 3D HOI reconstruction, we also propose several new tasks for 3D HOI understanding to pave the way for future work. Data and code will be publicly available at https://wenboran2002.github.io/3dhoi.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 50 of its 55 references that are in the map. First posted on arXiv 20 Mar 2025.

Builds on

50 of 55 references.

References · 50

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  2. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  3. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  4. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,612
  5. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 849
  6. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusioncited by 689
  7. 2023ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depthcited by 657
  8. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimizationcited by 560
  9. 2023Zero123++: a Single Image to Consistent Multi-view Diffusion Base Modelcited by 521
  10. 2024InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Modelscited by 433
  11. 2023Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priorscited by 397
  12. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Modelcited by 378
38 more
  1. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  2. 2023RealFusion: 360deg Reconstruction of Any Object from a Single Imagecited by 306
  3. 2022BEHAVE: Dataset and Method for Tracking Human Object Interactionscited by 249
  4. 2017Learning to Detect Human-Object Interactionscited by 240
  5. 2022CLIFF: Carrying Location Information in Full Frames into Human Pose and Shape Estimationcited by 238
  6. 2015Visual Semantic Role Labelingcited by 190
  7. 2023GauHuman: Articulated Gaussian Splatting from Monocular Human Videoscited by 168
  8. 2023SMPLer-X: Scaling Up Expressive Human Pose and Shape Estimationcited by 155
  9. 2022PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Imagescited by 153
  10. 2020Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wildcited by 152
  11. 2020Coherent Reconstruction of Multiple Humans from a Single Imagecited by 136
  12. 2023One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformercited by 130
  13. 2022CHORE: Contact, Human and Object REconstruction from a single RGB imagecited by 95
  14. 2020Detailed 2D-3D Joint Representation for Human-Object Interactioncited by 93
  15. 2022InterCap: Joint Markerless 3D Tracking of Humans and Objects in Interactioncited by 93
  16. 2022Compositional Human-Scene Interaction Synthesis with Semantic Controlcited by 83
  17. 2020PaStaNet: Toward Human Activity Knowledge Enginecited by 80
  18. 2023ChatPose: Chatting about 3D Human Posecited by 72
  19. 2023GPT4Point: A Unified Framework for Point-Language Understanding and Generationcited by 71
  20. 2023Exploring Predicate Visual Context in Detecting Human-Object Interactionscited by 56
  21. 2023LEMON: Learning 3D Human-Object Interaction Relation from 2D Imagescited by 49
  22. 2023Amodal Ground Truth and Completion in the Wildcited by 45
  23. 2023I'M HOI: Inertia-aware Monocular Capture of 3D Human-Object Interactionscited by 44
  24. 2022Reconstructing Action-Conditioned Human-Object Interactions Using Commonsense Knowledge Priorscited by 30
  25. 2022HAKE: A Knowledge Engine Foundation for Human Activity Understandingcited by 28
  26. 20246DGS: 6D Pose Estimation from a Single Image and a 3D Gaussian Splatting Modelcited by 28
  27. 2022OcclusionFusion: Occlusion-aware Motion Estimation for Real-time Dynamic 3D Reconstructioncited by 27
  28. 2024ImageNet3D: Towards General-Purpose Object-Level 3D Understandingcited by 25
  29. 2024F-HOI: Toward Fine-grained Semantic-Aligned 3D Human-Object Interactionscited by 24
  30. 2023From Isolated Islands to Pangea: Unifying Semantic Space for Human Action Understandingcited by 19
  31. 2024HumanVLA: Towards Vision-Language Directed Object Rearrangement by Physical Humanoidcited by 19
  32. 2024GS-Pose: Generalizable Segmentation-based 6D Object Pose Estimation with 3D Gaussian Splattingcited by 15
  33. 2023Bridging the Gap between Human Motion and Action Semantics via Kinematic Phrasescited by 13
  34. 2024Monocular Human-Object Reconstruction in the Wildcited by 9
  35. 2024ImDy: Human Inverse Dynamics from Imitated Observationscited by 9
  36. 2023Primitive-based 3D Human-Object Interaction Modelling and Programmingcited by 7
  37. 2017Care about you: towards large-scale human-centric visual relationship detectioncited by 6
  38. 2024Interacted Object Grounding in Spatio-Temporal Human-Object Interactionscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 7.

13 papers.

Sort:

By area:

  1. 2025CARI4D: Category Agnostic 4D Reconstruction of Human-Object Interactioncited by 14
  2. 2025TOUCH: Text-guided Controllable Generation of Free-Form Hand-Object Interactionscited by 8
  3. 2025GenHOI: Generalizing Text-driven 4D Human-Object Interaction Synthesis for Unseen Objectscited by 7
  4. 2025Advances in 4D Representation: Geometry, Motion, and Interactioncited by 4
  5. 2026ArtHOI: Taming Foundation Models for Monocular 4D Reconstruction of Hand-Articulated-Object Interactionscited by 4
  6. 2025Efficient and Scalable Monocular Human-Object Interaction Motion Reconstructioncited by 3
  7. 2026TeHOR: Text-Guided 3D Human and Object Reconstruction with Texturescited by 1
  8. 2026End-to-End Spatial-Temporal Transformer for Real-time 4D HOI Reconstructioncited by 1
  9. 2025Asset-Driven Sematic Reconstruction of Dynamic Scene with Multi-Human-Object Interactionscited by 0
  10. 2026LEXIS: LatEnt ProXimal Interaction Signatures for 3D HOI from an Imagecited by 0
  11. 2026Recovering Physically Plausible Human-Object Interactions from Monocular Videoscited by 0
  12. 2026Reconstructing Humans and Objects in Interaction using Large Reconstruction Modelscited by 0
1 more
  1. 2026Prosthesis-Aware 3D Human Pose Estimation: A Dataset and Benchmark for RSP Userscited by 0