# Joint Reconstruction of 3D Human and Object via Contact-Based Refinement Transformer

arXiv: https://arxiv.org/abs/2404.04819 · HTML: https://mlsanity.com/paper/2404.04819/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 56 references that are in the map.
- First posted on arXiv 7 Apr 2024.

## Builds on (most cited first, 30 of 46)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop](https://mlsanity.com/paper/1909.12828/) (2019)
- [VIBE: Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/1912.05656/) (2019)
- [PVNet: Pixel-wise Voting Network for 6DoF Pose Estimation](https://mlsanity.com/paper/1812.11788/) (2018)
- [DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion](https://mlsanity.com/paper/1901.04780/) (2019)
- [PARE: Part Attention Regressor for 3D Human Body Estimation](https://mlsanity.com/paper/2104.08527/) (2021)
- [HuMoR: 3D Human Motion Model for Robust Pose Estimation](https://mlsanity.com/paper/2105.04668/) (2021)
- [Convolutional Mesh Regression for Single-Image Human Shape Reconstruction](https://mlsanity.com/paper/1905.03244/) (2019)
- [Generating 3D faces using Convolutional Mesh Autoencoders](https://mlsanity.com/paper/1807.10267/) (2018)
- [Integral Human Pose Regression](https://mlsanity.com/paper/1711.08229/) (2017)
- [PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop](https://mlsanity.com/paper/2103.16507/) (2021)
- [BEHAVE: Dataset and Method for Tracking Human Object Interactions](https://mlsanity.com/paper/2204.06950/) (2022)
- [CLIFF: Carrying Location Information in Full Frames into Human Pose and Shape Estimation](https://mlsanity.com/paper/2208.00571/) (2022)
- [Real-Time Seamless Single Shot 6D Object Pose Prediction](https://mlsanity.com/paper/1711.08848/) (2017)
- [Capturing and Inferring Dense Full-Body Human-Scene Contact](https://mlsanity.com/paper/2206.09553/) (2022)
- [Populating 3D Scenes by Learning Human-Scene Interaction](https://mlsanity.com/paper/2012.11581/) (2020)
- [Beyond Static Features for Temporally Consistent 3D Human Pose and Shape from a Video](https://mlsanity.com/paper/2011.08627/) (2020)
- [Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild](https://mlsanity.com/paper/2007.15649/) (2020)
- [ZebraPose: Coarse to Fine Surface Encoding for 6DoF Object Pose Estimation](https://mlsanity.com/paper/2203.09418/) (2022)
- [Contact and Human Dynamics from Monocular Video](https://mlsanity.com/paper/2007.11678/) (2020)
- [CHORE: Contact, Human and Object REconstruction from a single RGB image](https://mlsanity.com/paper/2204.02445/) (2022)
- [InterCap: Joint Markerless 3D Tracking of Humans and Objects in Interaction](https://mlsanity.com/paper/2209.12354/) (2022)
- [On Self-Contact and Human Pose](https://mlsanity.com/paper/2104.03176/) (2021)

## Built on by (most cited first, 29 of 29)

- [CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangement](https://mlsanity.com/paper/2406.19353/) (2024)
- [HUMOTO: A 4D Dataset of Mocap Human Object Interactions](https://mlsanity.com/paper/2504.10414/) (2025)
- [EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Views](https://mlsanity.com/paper/2405.13659/) (2024)
- [InterTrack: Tracking Human Object Interaction without Object Templates](https://mlsanity.com/paper/2408.13953/) (2024)
- [PICO: Reconstructing 3D People In Contact with Objects](https://mlsanity.com/paper/2504.17695/) (2025)
- [InteractVLM: 3D Interaction Reasoning from 2D Foundational Models](https://mlsanity.com/paper/2504.05303/) (2025)
- [TriDi: Trilateral Diffusion of 3D Humans, Objects, and Interactions](https://mlsanity.com/paper/2412.06334/) (2024)
- [Learning Dense Hand Contact Estimation from Imbalanced Data](https://mlsanity.com/paper/2505.11152/) (2025)
- [CARI4D: Category Agnostic 4D Reconstruction of Human-Object Interaction](https://mlsanity.com/paper/2512.11988/) (2025)
- [ScoreHOI: Physically Plausible Reconstruction of Human-Object Interaction via Score-Guided Diffusion](https://mlsanity.com/paper/2509.07920/) (2025)
- [End-to-End HOI Reconstruction Transformer with Graph-based Encoding](https://mlsanity.com/paper/2503.06012/) (2025)
- [ECHO: Ego-Centric modeling of Human-Object interactions](https://mlsanity.com/paper/2508.21556/) (2025)
- [MMHOI: Modeling Complex 3D Multi-Human Multi-Object Interactions](https://mlsanity.com/paper/2510.07828/) (2025)
- [HOPE: Hand-Object Pressure Estimation from Monocular Videos](https://mlsanity.com/paper/2608.06192/) (2026)
- [Contact-Aware Amodal Completion for Human-Object Interaction via Multi-Regional Inpainting](https://mlsanity.com/paper/2508.00427/) (2025)
- [Shoe Style-Invariant and Ground-Aware Learning for Dense Foot Contact Estimation](https://mlsanity.com/paper/2511.22184/) (2025)
- [TeHOR: Text-Guided 3D Human and Object Reconstruction with Textures](https://mlsanity.com/paper/2602.19679/) (2026)
- [End-to-End Spatial-Temporal Transformer for Real-time 4D HOI Reconstruction](https://mlsanity.com/paper/2603.14435/) (2026)
- [GraphiContact: Pose-aware Human-Scene Robust Contact Perception for Interactive Systems](https://mlsanity.com/paper/2603.20310/) (2026)
- [Realistic Clothed Human and Object Joint Reconstruction from a Single Image](https://mlsanity.com/paper/2502.18150/) (2025)
- [CoopDiff: Anticipating 3D Human-object Interactions via Contact-consistent Decoupled Diffusion](https://mlsanity.com/paper/2508.07162/) (2025)
- [CRISP: Contact-Guided Real2Sim from Monocular Video with Planar Scene Primitives](https://mlsanity.com/paper/2512.14696/) (2025)
- [Learning Human-Object Interaction for 3D Human Pose Estimation from LiDAR Point Clouds](https://mlsanity.com/paper/2603.16343/) (2026)
- [Pi-HOC: Pairwise 3D Human-Object Contact Estimation](https://mlsanity.com/paper/2604.12923/) (2026)
- [LEXIS: LatEnt ProXimal Interaction Signatures for 3D HOI from an Image](https://mlsanity.com/paper/2604.20800/) (2026)
- [MaMi-HOI: Harmonizing Global Kinematics and Local Geometry for Human-Object Interaction Generation](https://mlsanity.com/paper/2605.05756/) (2026)
- [RHINO: Reconstructing Human Interactions with Novel Objects from Monocular Videos](https://mlsanity.com/paper/2605.17014/) (2026)
- [Reconstructing Humans and Objects in Interaction using Large Reconstruction Models](https://mlsanity.com/paper/2608.27407/) (2026)
- [GraspHOI: Full-Body 3D Human-Object Reconstruction with Finger-Level Grasps from a Single In-the-Wild Image](https://mlsanity.com/paper/2608.28386/) (2026)

## Abstract (arXiv)

Human-object contact serves as a strong cue to understand how humans physically interact with objects. Nevertheless, it is not widely explored to utilize human-object contact information for the joint reconstruction of 3D human and object from a single image. In this work, we present a novel joint 3D human-object reconstruction method (CONTHO) that effectively exploits contact information between humans and objects. There are two core designs in our system: 1) 3D-guided contact estimation and 2) contact-based 3D human and object refinement. First, for accurate human-object contact estimation, CONTHO initially reconstructs 3D humans and objects and utilizes them as explicit 3D guidance for contact estimation. Second, to refine the initial reconstructions of 3D human and object, we propose a novel contact-based refinement Transformer that effectively aggregates human features and object features based on the estimated human-object contact. The proposed contact-based refinement prevents the learning of erroneous correlation between human and object, which enables accurate 3D reconstruction. As a result, our CONTHO achieves state-of-the-art performance in both human-object contact estimation and joint reconstruction of 3D human and object. The code is publicly available at https://github.com/dqj5182/CONTHO_RELEASE.