mlsanityData to 7 Oct 2026

Detecting Human-Object Contact in Images

arXiv 2303.03373 ↗First posted 6 Mar 2023cs.CVcited by 41

Not on the map. Why?

Abstract (arXiv)

Humans constantly contact objects to move and perform tasks. Thus, detecting human-object contact is important for building human-centered artificial intelligence. However, there exists no robust method to detect contact between the body and the scene from an image, and there exists no dataset to learn such a detector. We fill this gap with HOT ("Human-Object conTact"), a new dataset of human-object contacts for images. To build HOT, we use two data sources: (1) We use the PROX dataset of 3D human meshes moving in 3D scenes, and automatically annotate 2D image areas for contact via 3D mesh proximity and projection. (2) We use the V-COCO, HAKE and Watch-n-Patch datasets, and ask trained annotators to draw polygons for the 2D image areas where contact takes place. We also annotate the involved body part of the human body. We use our HOT dataset to train a new contact detector, which takes a single color image as input, and outputs 2D contact heatmaps as well as the body-part labels that are in contact. This is a new and challenging task that extends current foot-ground or hand-object contact detectors to the full generality of the whole body. The detector uses a part-attention branch to guide contact estimation through the context of the surrounding body parts and scene. We evaluate our detector extensively, and quantitative results show that our model outperforms baselines, and that all components contribute to better performance. Results on images from an online repository show reasonable detections and generalizability.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Builds on 51 of its 71 references that are in the map. First posted on arXiv 6 Mar 2023.

Builds on

51 of 71 references.

References · 51

  1. 2016Pyramid Scene Parsing Networkcited by 2,472
  2. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  3. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  4. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  5. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,161
  6. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  7. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  8. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 397
  9. 2019Learning joint reconstruction of hands and manipulated objectscited by 377
  10. 2021PARE: Part Attention Regressor for 3D Human Body Estimationcited by 334
  11. 2021HuMoR: 3D Human Motion Model for Robust Pose Estimationcited by 319
  12. 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 306
38 more
  1. 2019Resolving 3D Human Pose Ambiguities with 3D Scene Constraintscited by 294
  2. 2017Dilated Residual Networkscited by 293
  3. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
  4. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  5. 2022BEHAVE: Dataset and Method for Tracking Human Object Interactionscited by 246
  6. 2023Diffusion-based Generation, Optimization, and Planning in 3D Scenescited by 237
  7. 2015Visual Semantic Role Labelingcited by 188
  8. 2020ContactPose: A Dataset of Grasps with Object Contact and Hand Posecited by 180
  9. 2018Learning Human-Object Interactions by Graph Parsing Neural Networkscited by 176
  10. 2022Capturing and Inferring Dense Full-Body Human-Scene Contactcited by 176
  11. 2021FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integrationcited by 168
  12. 2020Populating 3D Scenes by Learning Human-Scene Interactioncited by 164
  13. 2020Reconstructing Hand-Object Interactions in the Wildcited by 143
  14. 2021Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Timecited by 140
  15. 2021ContactOpt: Optimizing Contact to Improve Graspscited by 137
  16. 2018Grounded Human-Object Interaction Hotspots from Videocited by 128
  17. 2019Generating 3D People in Scenes without Peoplecited by 127
  18. 20213D AffordanceNet: A Benchmark for Visual Object Affordance Understandingcited by 100
  19. 2021Learning Motion Priors for 4D Human Body Capture in 3D Scenescited by 98
  20. 2022CHORE: Contact, Human and Object REconstruction from a single RGB imagecited by 95
  21. 2020Contact and Human Dynamics from Monocular Videocited by 94
  22. 2020State-Only Imitation Learning for Dexterous Manipulationcited by 90
  23. 2020PaStaNet: Toward Human Activity Knowledge Enginecited by 79
  24. 2022Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applicationscited by 78
  25. 2019Holistic++ Scene Understanding: Single-view 3D Holistic Scene Parsing and Human Pose Estimation with Human-Object Interaction and Physical Commonsensecited by 71
  26. 2020Learning Affordance Landscapes for Interaction Exploration in 3D Environmentscited by 65
  27. 2022MIME: Human-Aware 3D Scene Generationcited by 57
  28. 2019Deep Contextual Attention for Human-Object Interaction Detectioncited by 54
  29. 2020Holistic 3D Human and Scene Mesh Estimation from Single View Imagescited by 54
  30. 2021Home Action Genome: Cooperative Compositional Action Understandingcited by 52
  31. 2019Estimating 3D Motion and Forces of Person-Object Interactions from Monocular Videocited by 48
  32. 2020LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activitiescited by 46
  33. 2020Detecting Hands and Recognizing Physical Contact in the Wildcited by 38
  34. 2020Bodies at Rest: 3D Human Pose and Shape Estimation from a Pressure Image using Synthetic Datacited by 34
  35. 2022HULC: 3D Human Motion Capture with Pose Manifold Sampling and Dense Contact Guidancecited by 29
  36. 2017Baxter's Homunculus: Virtual Reality Spaces for Teleoperation in Manufacturingcited by 19
  37. 2022MoCapDeform: Monocular 3D Human Motion Capture in Deformable Scenescited by 18
  38. 2022Visual Haptic Reasoning: Estimating Contact Forces by Observing Deformable Object Interactionscited by 5

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 10; 2025: 13; 2026: 9.

41 papers.

Sort:

By area:

  1. 2023InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusioncited by 181
  2. 20233D Human Pose Estimation via Intuitive Physicscited by 100
  3. 2024Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordancecited by 79
  4. 2023PhysHOI: Physics-Based Imitation of Dynamic Human-Object Interactioncited by 72
  5. 2023DECO: Dense Estimation of 3D Human-Scene Contact In The Wildcited by 54
  6. 2023LEMON: Learning 3D Human-Object Interaction Relation from 2D Imagescited by 49
  7. 2024PhyRecon: Physically Plausible Neural Scene Reconstructioncited by 44
  8. 2023Template Free Reconstruction of Human-object Interaction with Procedural Interaction Generationcited by 37
  9. 2024Joint Reconstruction of 3D Human and Object via Contact-Based Refinement Transformercited by 29
  10. 2024EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Viewscited by 26
  11. 2025PICO: Reconstructing 3D People In Contact with Objectscited by 22
  12. 2025InteractVLM: 3D Interaction Reasoning from 2D Foundational Modelscited by 21
29 more
  1. 2024TriDi: Trilateral Diffusion of 3D Humans, Objects, and Interactionscited by 18
  2. 2024MMVP: A Multimodal MoCap Dataset with Vision and Pressure Sensorscited by 16
  3. 2023Learning Explicit Contact for Implicit Reconstruction of Hand-held Objects from Monocular Imagescited by 10
  4. 2024PEAR: Phrase-Based Hand-Object Interaction Anticipationcited by 10
  5. 2025End-to-End HOI Reconstruction Transformer with Graph-based Encodingcited by 8
  6. 2023Ins-HOI: Instance Aware Human-Object Interactions Recoverycited by 7
  7. 2024Precision-Enhanced Human-Object Contact Detection via Depth-Aware Perspective Interaction and Object Texture Restorationcited by 5
  8. 2025Re-Aligning Language to Visual Objects with an Agentic Workflowcited by 4
  9. 2025VHOI: Controllable Video Generation of Human-Object Interactions from Sparse Trajectories via Motion Densificationcited by 4
  10. 2026FEEL (Force-Enhanced Egocentric Learning): A Dataset for Physical Action Understandingcited by 4
  11. 2025GRACE: Estimating Geometry-level 3D Human-Scene Contact from 2D Imagescited by 3
  12. 2023BPJDet: Extended Object Representation for Generic Body-Part Joint Detectioncited by 2
  13. 2024Visual-Geometric Collaborative Guidance for Affordance Learningcited by 2
  14. 2025Generative Modeling of Shape-Dependent Self-Contact Human Posescited by 2
  15. 2025Decoupled Generative Modeling for Human-Object Interaction Synthesiscited by 2
  16. 2026Detecting Precise Hand Touch Moments in Egocentric Videocited by 2
  17. 2024Ask, Pose, Unite: Scaling Data Acquisition for Close Interactions with Vision Language Modelscited by 1
  18. 2025Prompt Guidance and Human Proximal Perception for HOT Prediction with Regional Joint Losscited by 1
  19. 2025Contact-Aware Amodal Completion for Human-Object Interaction via Multi-Regional Inpaintingcited by 1
  20. 2025DecoDINO: 3D Human-Scene Contact Prediction with Semantic Classificationcited by 1
  21. 2026TeHOR: Text-Guided 3D Human and Object Reconstruction with Texturescited by 1
  22. 2026GraphiContact: Pose-aware Human-Scene Robust Contact Perception for Interactive Systemscited by 1
  23. 2026InterPhys: Physics-aware Human Motion Synthesis in a Dynamic Scenecited by 1
  24. 2025What-Meets-Where: Unified Learning of Action and Contact Localization in Imagescited by 0
  25. 2025TMUAD: Enhancing Logical Capabilities in Unified Anomaly Detection Models with a Text Memory Bankcited by 0
  26. 2026Pi-HOC: Pairwise 3D Human-Object Contact Estimationcited by 0
  27. 2026LEXIS: LatEnt ProXimal Interaction Signatures for 3D HOI from an Imagecited by 0
  28. 2026Efficient Human-Contact Representation for Human-Scene Interactioncited by 0
  29. 2026EgoPHI: Estimating 3D Hand-Object Contact and Force from Egocentric Visioncited by 0