# End-to-End HOI Reconstruction Transformer with Graph-based Encoding arXiv: https://arxiv.org/abs/2503.06012 ยท HTML: https://mlsanity.com/paper/2503.06012/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 36 of its 47 references that are in the map. - First posted on arXiv 8 Mar 2025. ## Builds on (most cited first, 30 of 36) - [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017) - [End-to-End Human Pose and Mesh Reconstruction with Transformers](https://mlsanity.com/paper/2012.09760/) (2020) - [HuMoR: 3D Human Motion Model for Robust Pose Estimation](https://mlsanity.com/paper/2105.04668/) (2021) - [HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation](https://mlsanity.com/paper/2011.14672/) (2020) - [I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image](https://mlsanity.com/paper/2008.03713/) (2020) - [Convolutional Mesh Regression for Single-Image Human Shape Reconstruction](https://mlsanity.com/paper/1905.03244/) (2019) - [Generating 3D faces using Convolutional Mesh Autoencoders](https://mlsanity.com/paper/1807.10267/) (2018) - [Mesh Graphormer](https://mlsanity.com/paper/2104.00272/) (2021) - [BEHAVE: Dataset and Method for Tracking Human Object Interactions](https://mlsanity.com/paper/2204.06950/) (2022) - [Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose](https://mlsanity.com/paper/2008.09047/) (2020) - [CLIFF: Carrying Location Information in Full Frames into Human Pose and Shape Estimation](https://mlsanity.com/paper/2208.00571/) (2022) - [Neural Body Fitting: Unifying Deep Learning and Model-Based Human Pose and Shape Estimation](https://mlsanity.com/paper/1808.05942/) (2018) - [Capturing and Inferring Dense Full-Body Human-Scene Contact](https://mlsanity.com/paper/2206.09553/) (2022) - [Monocular Expressive Body Regression through Body-Driven Attention](https://mlsanity.com/paper/2008.09062/) (2020) - [SMPLer-X: Scaling Up Expressive Human Pose and Shape Estimation](https://mlsanity.com/paper/2309.17448/) (2023) - [Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild](https://mlsanity.com/paper/2007.15649/) (2020) - [Collaborative Regression of Expressive Bodies using Moderation](https://mlsanity.com/paper/2105.05301/) (2021) - [One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformer](https://mlsanity.com/paper/2303.16160/) (2023) - [CHORE: Contact, Human and Object REconstruction from a single RGB image](https://mlsanity.com/paper/2204.02445/) (2022) - [InterCap: Joint Markerless 3D Tracking of Humans and Objects in Interaction](https://mlsanity.com/paper/2209.12354/) (2022) - [Synthetic Training for Accurate 3D Human Pose and Shape Estimation in the Wild](https://mlsanity.com/paper/2009.10013/) (2020) - [Accurate 3D Hand Pose Estimation for Whole-Body 3D Human Mesh Estimation](https://mlsanity.com/paper/2011.11534/) (2020) - [Cross-Attention of Disentangled Modalities for 3D Human Mesh Recovery with Transformers](https://mlsanity.com/paper/2207.13820/) (2022) - [NeuralDome: A Neural Modeling Pipeline on Multi-View Human-Object Interactions](https://mlsanity.com/paper/2212.07626/) (2022) - [Visibility Aware Human-Object Interaction Tracking from Single RGB Camera](https://mlsanity.com/paper/2303.16479/) (2023) - [Holistic 3D Human and Scene Mesh Estimation from Single View Images](https://mlsanity.com/paper/2012.01591/) (2020) - [DECO: Dense Estimation of 3D Human-Scene Contact In The Wild](https://mlsanity.com/paper/2309.15273/) (2023) - [Detecting Human-Object Contact in Images](https://mlsanity.com/paper/2303.03373/) (2023) - [Learning Complex 3D Human Self-Contact](https://mlsanity.com/paper/2012.10366/) (2020) - [POTTER: Pooling Attention Transformer for Efficient Human Mesh Recovery](https://mlsanity.com/paper/2303.13357/) (2023) ## Built on by (most cited first, 8 of 8) - [Advances in 4D Representation: Geometry, Motion, and Interaction](https://mlsanity.com/paper/2510.19255/) (2025) - [VHOI: Controllable Video Generation of Human-Object Interactions from Sparse Trajectories via Motion Densification](https://mlsanity.com/paper/2512.09646/) (2025) - [TeHOR: Text-Guided 3D Human and Object Reconstruction with Textures](https://mlsanity.com/paper/2602.19679/) (2026) - [End-to-End Spatial-Temporal Transformer for Real-time 4D HOI Reconstruction](https://mlsanity.com/paper/2603.14435/) (2026) - [LEXIS: LatEnt ProXimal Interaction Signatures for 3D HOI from an Image](https://mlsanity.com/paper/2604.20800/) (2026) - [Real2Sim in HOI: Toward Physically Plausible HOI Reconstruction from Monocular Videos](https://mlsanity.com/paper/2605.14462/) (2026) - [Reconstructing Humans and Objects in Interaction using Large Reconstruction Models](https://mlsanity.com/paper/2608.27407/) (2026) - [GraspHOI: Full-Body 3D Human-Object Reconstruction with Finger-Level Grasps from a Single In-the-Wild Image](https://mlsanity.com/paper/2608.28386/) (2026) ## Abstract (arXiv) With the diversification of human-object interaction (HOI) applications and the success of capturing human meshes, HOI reconstruction has gained widespread attention. Existing mainstream HOI reconstruction methods often rely on explicitly modeling interactions between humans and objects. However, such a way leads to a natural conflict between 3D mesh reconstruction, which emphasizes global structure, and fine-grained contact reconstruction, which focuses on local details. To address the limitations of explicit modeling, we propose the End-to-End HOI Reconstruction Transformer with Graph-based Encoding (HOI-TG). It implicitly learns the interaction between humans and objects by leveraging self-attention mechanisms. Within the transformer architecture, we devise graph residual blocks to aggregate the topology among vertices of different spatial structures. This dual focus effectively balances global and local representations. Without bells and whistles, HOI-TG achieves state-of-the-art performance on BEHAVE and InterCap datasets. Particularly on the challenging InterCap dataset, our method improves the reconstruction results for human and object meshes by 8.9% and 8.6%, respectively.