# DECO: Dense Estimation of 3D Human-Scene Contact In The Wild

arXiv: https://arxiv.org/abs/2309.15273 · HTML: https://mlsanity.com/paper/2309.15273/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Builds on 67 of its 91 references that are in the map.
- First posted on arXiv 26 Sep 2023.

## Builds on (most cited first, 30 of 67)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop](https://mlsanity.com/paper/1909.12828/) (2019)
- [ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation](https://mlsanity.com/paper/2204.12484/) (2022)
- [End-to-End Human Pose and Mesh Reconstruction with Transformers](https://mlsanity.com/paper/2012.09760/) (2020)
- [GRAB: A Dataset of Whole-Body Human Grasping of Objects](https://mlsanity.com/paper/2008.11200/) (2020)
- [Learning joint reconstruction of hands and manipulated objects](https://mlsanity.com/paper/1904.05767/) (2019)
- [PARE: Part Attention Regressor for 3D Human Body Estimation](https://mlsanity.com/paper/2104.08527/) (2021)
- [HuMoR: 3D Human Motion Model for Robust Pose Estimation](https://mlsanity.com/paper/2105.04668/) (2021)
- [Resolving 3D Human Pose Ambiguities with 3D Scene Constraints](https://mlsanity.com/paper/1908.06963/) (2019)
- [Understanding Human Hands in Contact at Internet Scale](https://mlsanity.com/paper/2006.06669/) (2020)
- [PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop](https://mlsanity.com/paper/2103.16507/) (2021)
- [BEHAVE: Dataset and Method for Tracking Human Object Interactions](https://mlsanity.com/paper/2204.06950/) (2022)
- [CLIFF: Carrying Location Information in Full Frames into Human Pose and Shape Estimation](https://mlsanity.com/paper/2208.00571/) (2022)
- [Long-term Human Motion Prediction with Scene Context](https://mlsanity.com/paper/2007.03672/) (2020)
- [Visual Semantic Role Labeling](https://mlsanity.com/paper/1505.04474/) (2015)
- [ContactPose: A Dataset of Grasps with Object Contact and Hand Pose](https://mlsanity.com/paper/2007.09545/) (2020)
- [HOTR: End-to-End Human-Object Interaction Detection with Transformers](https://mlsanity.com/paper/2104.13682/) (2021)
- [Capturing and Inferring Dense Full-Body Human-Scene Contact](https://mlsanity.com/paper/2206.09553/) (2022)
- [Learning Human-Object Interactions by Graph Parsing Neural Networks](https://mlsanity.com/paper/1808.07962/) (2018)

## Built on by (most cited first, 30 of 54)

- [Scaling Up Dynamic Human-Scene Interaction Modeling](https://mlsanity.com/paper/2403.08629/) (2024)
- [PhysHOI: Physics-Based Imitation of Dynamic Human-Object Interaction](https://mlsanity.com/paper/2312.04393/) (2023)
- [LEMON: Learning 3D Human-Object Interaction Relation from 2D Images](https://mlsanity.com/paper/2312.08963/) (2023)
- [Template Free Reconstruction of Human-object Interaction with Procedural Interaction Generation](https://mlsanity.com/paper/2312.07063/) (2023)
- [Beyond the Contact: Discovering Comprehensive Affordance for 3D Objects from Pre-trained 2D Diffusion Models](https://mlsanity.com/paper/2401.12978/) (2024)
- [SMPLest-X: Ultimate Scaling for Expressive Human Pose and Shape Estimation](https://mlsanity.com/paper/2501.09782/) (2025)
- [Harmony4D: A Video Dataset for In-The-Wild Close Human Interactions](https://mlsanity.com/paper/2410.20294/) (2024)
- [Joint Reconstruction of 3D Human and Object via Contact-Based Refinement Transformer](https://mlsanity.com/paper/2404.04819/) (2024)
- [EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Views](https://mlsanity.com/paper/2405.13659/) (2024)
- [PICO: Reconstructing 3D People In Contact with Objects](https://mlsanity.com/paper/2504.17695/) (2025)
- [InteractVLM: 3D Interaction Reasoning from 2D Foundational Models](https://mlsanity.com/paper/2504.05303/) (2025)
- [TriDi: Trilateral Diffusion of 3D Humans, Objects, and Interactions](https://mlsanity.com/paper/2412.06334/) (2024)
- [MMVP: A Multimodal MoCap Dataset with Vision and Pressure Sensors](https://mlsanity.com/paper/2403.17610/) (2024)
- [Contact-aware Human Motion Generation from Textual Descriptions](https://mlsanity.com/paper/2403.15709/) (2024)
- [Learning Dense Hand Contact Estimation from Imbalanced Data](https://mlsanity.com/paper/2505.11152/) (2025)
- [CARI4D: Category Agnostic 4D Reconstruction of Human-Object Interaction](https://mlsanity.com/paper/2512.11988/) (2025)
- [InteractAnything: Zero-shot Human Object Interaction Synthesis via LLM Feedback and Object Affordance Parsing](https://mlsanity.com/paper/2505.24315/) (2025)
- [ChatHuman: Chatting about 3D Humans with Tools](https://mlsanity.com/paper/2405.04533/) (2024)
- [How Do I Do That? Synthesizing 3D Hand Motion and Contacts for Everyday Interactions](https://mlsanity.com/paper/2504.12284/) (2025)
- [End-to-End HOI Reconstruction Transformer with Graph-based Encoding](https://mlsanity.com/paper/2503.06012/) (2025)
- [ECHO: Ego-Centric modeling of Human-Object interactions](https://mlsanity.com/paper/2508.21556/) (2025)
- [DAViD: Modeling Dynamic Affordance of 3D Objects Using Pre-trained Video Diffusion Models](https://mlsanity.com/paper/2501.08333/) (2025)
- [FEEL (Force-Enhanced Egocentric Learning): A Dataset for Physical Action Understanding](https://mlsanity.com/paper/2603.15847/) (2026)
- [Geometric Visual Fusion Graph Neural Networks for Multi-Person Human-Object Interaction Recognition in Videos](https://mlsanity.com/paper/2506.03440/) (2025)
- [Efficient and Scalable Monocular Human-Object Interaction Motion Reconstruction](https://mlsanity.com/paper/2512.00960/) (2025)
- [Decoupled Generative Modeling for Human-Object Interaction Synthesis](https://mlsanity.com/paper/2512.19049/) (2025)
- [Towards in-the-wild Egocentric 3D Hand-Object Pose Estimation](https://mlsanity.com/paper/2606.30598/) (2026)
- [HOPE: Hand-Object Pressure Estimation from Monocular Videos](https://mlsanity.com/paper/2608.06192/) (2026)
- [EgoTac: In-the-wild Tactile Prediction from Egocentric Vision](https://mlsanity.com/paper/2608.15060/) (2026)
- [Kinematics-based 3D Human-Object Interaction Reconstruction from Single View](https://mlsanity.com/paper/2407.14043/) (2024)

## Abstract (arXiv)

Understanding how humans use physical contact to interact with the world is key to enabling human-centric artificial intelligence. While inferring 3D contact is crucial for modeling realistic and physically-plausible human-object interactions, existing methods either focus on 2D, consider body joints rather than the surface, use coarse 3D body regions, or do not generalize to in-the-wild images. In contrast, we focus on inferring dense, 3D contact between the full body surface and objects in arbitrary images. To achieve this, we first collect DAMON, a new dataset containing dense vertex-level contact annotations paired with RGB images containing complex human-object and human-scene contact. Second, we train DECO, a novel 3D contact detector that uses both body-part-driven and scene-context-driven attention to estimate vertex-level contact on the SMPL body. DECO builds on the insight that human observers recognize contact by reasoning about the contacting body parts, their proximity to scene objects, and the surrounding scene context. We perform extensive evaluations of our detector on DAMON as well as on the RICH and BEHAVE datasets. We significantly outperform existing SOTA methods across all benchmarks. We also show qualitatively that DECO generalizes well to diverse and challenging real-world human interactions in natural images. The code, data, and models are available at https://deco.is.tue.mpg.de.