mlsanityData to 14 Oct 2026

Object pop-up: Can we infer 3D objects and their poses from human interactions alone?

arXiv 2306.00777 ↗First posted 1 Jun 2023cs.CVcited by 37

Not on the map. Why?

Abstract (arXiv)

The intimate entanglement between objects affordances and human poses is of large interest, among others, for behavioural sciences, cognitive psychology, and Computer Vision communities. In recent years, the latter has developed several object-centric approaches: starting from items, learning pipelines synthesizing human poses and dynamics in a realistic way, satisfying both geometrical and functional expectations. However, the inverse perspective is significantly less explored: Can we infer 3D objects and their poses from human interactions alone? Our investigation follows this direction, showing that a generic 3D human point cloud is enough to pop up an unobserved object, even when the user is just imitating a functionality (e.g., looking through a binocular) without involving a tangible counterpart. We validate our method qualitatively and quantitatively, with synthetic data and sequences acquired for the task, showing applicability for XR/VR. The code is available at https://github.com/ptrvilya/object-popup.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Builds on 53 of its 80 references that are in the map. First posted on arXiv 1 Jun 2023.

Builds on

53 of 80 references.

References · 53

  1. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  2. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  3. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,612
  4. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 405
  5. 2019Learning joint reconstruction of hands and manipulated objectscited by 378
  6. 2022HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interactioncited by 327
  7. 2021DexYCB: A Benchmark for Capturing Hand Grasping of Objectscited by 325
  8. 2019Resolving 3D Human Pose Ambiguities with 3D Scene Constraintscited by 295
  9. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 254
  10. 2022BEHAVE: Dataset and Method for Tracking Human Object Interactionscited by 249
  11. 2021Stochastic Scene-Aware Motion Predictioncited by 203
  12. 2020Long-term Human Motion Prediction with Scene Contextcited by 192
38 more
  1. 2020ContactPose: A Dataset of Grasps with Object Contact and Hand Posecited by 183
  2. 2022Embodied Hands: Modeling and Capturing Hands and Bodies Togethercited by 181
  3. 2022Capturing and Inferring Dense Full-Body Human-Scene Contactcited by 180
  4. 2020Grasping Field: Learning Implicit Representations for Human Graspscited by 179
  5. 2020Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wildcited by 152
  6. 2019ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imagingcited by 147
  7. 2020Reconstructing Hand-Object Interactions in the Wildcited by 144
  8. 2021GOAL: Generating 4D Whole-Body Motion for Hand-Object Graspingcited by 144
  9. 2021ContactOpt: Optimizing Contact to Improve Graspscited by 139
  10. 2021Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensorscited by 137
  11. 2022COUCH: Towards Controllable Human-Chair Interactionscited by 135
  12. 2020CPF: Learning a Contact Potential Field to Model the Hand-Object Interactioncited by 128
  13. 2021Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose Estimationcited by 123
  14. 2021D-Grasp: Physically Plausible Dynamic Grasp Synthesis for Hand-Object Interactionscited by 113
  15. 2018PointCloud Saliency Mapscited by 112
  16. 20213D AffordanceNet: A Benchmark for Visual Object Affordance Understandingcited by 100
  17. 2021Learning Motion Priors for 4D Human Body Capture in 3D Scenescited by 98
  18. 2022CHORE: Contact, Human and Object REconstruction from a single RGB imagecited by 95
  19. 2022InterCap: Joint Markerless 3D Tracking of Humans and Objects in Interactioncited by 93
  20. 2021SAGA: Stochastic Whole-Body Grasping with Contactcited by 91
  21. 2022What's in your hands? 3D Reconstruction of Generic Objects in Handscited by 90
  22. 2019ContactGrasp: Functional Multi-finger Grasp Synthesis from Contactcited by 87
  23. 2020Learning Dexterous Grasping with Object-Centric Visual Affordancescited by 85
  24. 2019Holistic++ Scene Understanding: Single-view 3D Holistic Scene Parsing and Human Pose Estimation with Human-Object Interaction and Physical Commonsensecited by 71
  25. 2022TOCH: Spatio-Temporal Object-to-Hand Correspondence for Motion Refinementcited by 70
  26. 2021Towards unconstrained joint hand-object reconstruction from RGB videoscited by 58
  27. 2023Visibility Aware Human-Object Interaction Tracking from Single RGB Cameracited by 58
  28. 2022Human-Aware Object Placement for Visual Environment Reconstructioncited by 56
  29. 2020Holistic 3D Human and Scene Mesh Estimation from Single View Imagescited by 54
  30. 2019Predicting 3D Human Dynamics from Videocited by 51
  31. 2022NeuralHOFusion: Neural Volumetric Rendering under Human-object Interactionscited by 44
  32. 2020Use the Force, Luke! Learning to Predict Physical Forces by Simulating Effectscited by 40
  33. 2021Neural Free-Viewpoint Performance Rendering under Complex Human-object Interactionscited by 37
  34. 2022Learned Vertex Descent: A New Direction for 3D Human Model Fittingcited by 37
  35. 2021Gravity-Aware Monocular 3D Human-Object Reconstructioncited by 30
  36. 2022Interaction Replica: Tracking Human-Object Interaction and Scene Changes From Human Motioncited by 27
  37. 2021RobustFusion: Robust Volumetric Performance Reconstruction under Human-object Interactions from Monocular RGBD Streamcited by 26
  38. 2021Pose2Room: Understanding 3D Scenes from Human Activitiescited by 22

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 10; 2024: 12; 2025: 7; 2026: 8.

37 papers.

Sort:

By area:

  1. 2023InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusioncited by 182
  2. 2023Controllable Human-Object Interaction Synthesiscited by 109
  3. 2023PhysHOI: Physics-Based Imitation of Dynamic Human-Object Interactioncited by 76
  4. 2023Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction Clipscited by 74
  5. 2025InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactionscited by 72
  6. 2024InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interactioncited by 64
  7. 2024ParaHome: Parameterizing Everyday Home Activities Towards 3D Generative Modeling of Human-Object Interactionscited by 57
  8. 2023LEMON: Learning 3D Human-Object Interaction Relation from 2D Imagescited by 49
  9. 2024THOR: Text to Human-Object Interaction Diffusion via Relation Interventioncited by 46
  10. 2023I'M HOI: Inertia-aware Monocular Capture of 3D Human-Object Interactionscited by 44
  11. 2024Human-3Diffusion: Realistic Avatar Creation via Explicit 3D Consistent Diffusion Modelscited by 44
  12. 2023Template Free Reconstruction of Human-object Interaction with Procedural Interaction Generationcited by 38
25 more
  1. 2024Beyond the Contact: Discovering Comprehensive Affordance for 3D Objects from Pre-trained 2D Diffusion Modelscited by 32
  2. 2025InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generationcited by 32
  3. 2023ROAM: Robust and Object-Aware Motion Generation Using Neural Pose Descriptorscited by 23
  4. 2024InterTrack: Tracking Human Object Interaction without Object Templatescited by 23
  5. 2023NSF: Neural Surface Fields for Human Modeling from Monocular Depthcited by 21
  6. 2024TriDi: Trilateral Diffusion of 3D Humans, Objects, and Interactionscited by 20
  7. 2024Gen-3Diffusion: Realistic Image-to-3D Generation via 2D & 3D Diffusion Synergycited by 16
  8. 2023Ins-HOI: Instance Aware Human-Object Interactions Recoverycited by 7
  9. 2024FORCE: Physics-aware Human-object Interactioncited by 7
  10. 2025GenHOI: Generalizing Text-driven 4D Human-Object Interaction Synthesis for Unseen Objectscited by 7
  11. 2026Unleashing Guidance Without Classifiers for Human-Object Interaction Animationcited by 6
  12. 2025ECHO: Ego-Centric modeling of Human-Object interactionscited by 5
  13. 2024COLLAGE: Collaborative Human-Agent Interaction Generation using Hierarchical Latent Diffusion and Language Modelscited by 3
  14. 2024Di2Pose: Discrete Diffusion Model for Occluded 3D Human Pose Estimationcited by 2
  15. 2025Category-Aware 3D Object Composition with Disentangled Texture and Shape Multi-view Diffusioncited by 2
  16. 2026Uni-HOI:A Unified framework for Learning the Joint distribution of Text and Human-Object Interactioncited by 2
  17. 2025H2OFlow: Grounding Human-Object Affordances with 3D Generative Models and Dense Diffused Flowscited by 1
  18. 2024GenHeld: Generating and Editing Handheld Objectscited by 0
  19. 2025CoopDiff: Anticipating 3D Human-object Interactions via Contact-consistent Decoupled Diffusioncited by 0
  20. 2026Physically Grounded 3D Generative Reconstruction under Hand Occlusion using Proprioception and Multi-Contact Touchcited by 0
  21. 2026SynAgent: Generalizable Cooperative Humanoid Manipulation via Solo-to-Cooperative Agent Synergycited by 0
  22. 2026IMU-HOI: A Symbiotic Framework for Coherent Human-Object Interaction and Motion Capture via Contact-Conscious Inertial Fusioncited by 0
  23. 2026Efficient Human-Contact Representation for Human-Scene Interactioncited by 0
  24. 2026GraspHOI: Full-Body 3D Human-Object Reconstruction with Finger-Level Grasps from a Single In-the-Wild Imagecited by 0
  25. 2026COSMI: COmpositional Synthesis of Multi-object Interactionscited by 0