mlsanityData to 14 Oct 2026

End-to-End HOI Reconstruction Transformer with Graph-based Encoding

arXiv 2503.06012 ↗First posted 8 Mar 2025cs.CVcited by 8

Not on the map. Why?

Abstract (arXiv)

With the diversification of human-object interaction (HOI) applications and the success of capturing human meshes, HOI reconstruction has gained widespread attention. Existing mainstream HOI reconstruction methods often rely on explicitly modeling interactions between humans and objects. However, such a way leads to a natural conflict between 3D mesh reconstruction, which emphasizes global structure, and fine-grained contact reconstruction, which focuses on local details. To address the limitations of explicit modeling, we propose the End-to-End HOI Reconstruction Transformer with Graph-based Encoding (HOI-TG). It implicitly learns the interaction between humans and objects by leveraging self-attention mechanisms. Within the transformer architecture, we devise graph residual blocks to aggregate the topology among vertices of different spatial structures. This dual focus effectively balances global and local representations. Without bells and whistles, HOI-TG achieves state-of-the-art performance on BEHAVE and InterCap datasets. Particularly on the challenging InterCap dataset, our method improves the reconstruction results for human and object meshes by 8.9% and 8.6%, respectively.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 36 of its 47 references that are in the map. First posted on arXiv 8 Mar 2025.

Builds on

36 of 47 references.

References · 36

  1. 2017End-to-end Recovery of Human Shape and Posecited by 885
  2. 2020End-to-End Human Pose and Mesh Reconstruction with Transformerscited by 418
  3. 2021HuMoR: 3D Human Motion Model for Robust Pose Estimationcited by 324
  4. 2020HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimationcited by 307
  5. 2020I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Imagecited by 274
  6. 2019Convolutional Mesh Regression for Single-Image Human Shape Reconstructioncited by 270
  7. 2018Generating 3D faces using Convolutional Mesh Autoencoderscited by 268
  8. 2021Mesh Graphormercited by 265
  9. 2022BEHAVE: Dataset and Method for Tracking Human Object Interactionscited by 249
  10. 2020Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Posecited by 248
  11. 2022CLIFF: Carrying Location Information in Full Frames into Human Pose and Shape Estimationcited by 238
  12. 2018Neural Body Fitting: Unifying Deep Learning and Model-Based Human Pose and Shape Estimationcited by 185
24 more
  1. 2022Capturing and Inferring Dense Full-Body Human-Scene Contactcited by 180
  2. 2020Monocular Expressive Body Regression through Body-Driven Attentioncited by 155
  3. 2023SMPLer-X: Scaling Up Expressive Human Pose and Shape Estimationcited by 155
  4. 2020Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wildcited by 152
  5. 2021Collaborative Regression of Expressive Bodies using Moderationcited by 137
  6. 2023One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformercited by 130
  7. 2022CHORE: Contact, Human and Object REconstruction from a single RGB imagecited by 95
  8. 2022InterCap: Joint Markerless 3D Tracking of Humans and Objects in Interactioncited by 93
  9. 2020Synthetic Training for Accurate 3D Human Pose and Shape Estimation in the Wildcited by 82
  10. 2020Accurate 3D Hand Pose Estimation for Whole-Body 3D Human Mesh Estimationcited by 81
  11. 2022Cross-Attention of Disentangled Modalities for 3D Human Mesh Recovery with Transformerscited by 81
  12. 2022NeuralDome: A Neural Modeling Pipeline on Multi-View Human-Object Interactionscited by 68
  13. 2023Visibility Aware Human-Object Interaction Tracking from Single RGB Cameracited by 58
  14. 2020Holistic 3D Human and Scene Mesh Estimation from Single View Imagescited by 54
  15. 2023DECO: Dense Estimation of 3D Human-Scene Contact In The Wildcited by 54
  16. 2023Detecting Human-Object Contact in Imagescited by 41
  17. 2020Learning Complex 3D Human Self-Contactcited by 40
  18. 2023POTTER: Pooling Attention Transformer for Efficient Human Mesh Recoverycited by 31
  19. 2022HULC: 3D Human Motion Capture with Pose Manifold Sampling and Dense Contact Guidancecited by 29
  20. 2024Joint Reconstruction of 3D Human and Object via Contact-Based Refinement Transformercited by 29
  21. 2023Sampling is Matter: Point-guided 3D Human Mesh Reconstructioncited by 24
  22. 2023Learning Explicit Contact for Implicit Reconstruction of Hand-held Objects from Monocular Imagescited by 10
  23. 2024RHOBIN Challenge: Reconstruction of Human Object Interactioncited by 10
  24. 2024StackFLOW: Monocular Human-Object Reconstruction by Stacked Normalizing Flow with Offsetcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 6.

8 papers.

  1. 2025Advances in 4D Representation: Geometry, Motion, and Interactioncited by 4
  2. 2025VHOI: Controllable Video Generation of Human-Object Interactions from Sparse Trajectories via Motion Densificationcited by 4
  3. 2026TeHOR: Text-Guided 3D Human and Object Reconstruction with Texturescited by 1
  4. 2026End-to-End Spatial-Temporal Transformer for Real-time 4D HOI Reconstructioncited by 1
  5. 2026LEXIS: LatEnt ProXimal Interaction Signatures for 3D HOI from an Imagecited by 0
  6. 2026Real2Sim in HOI: Toward Physically Plausible HOI Reconstruction from Monocular Videoscited by 0
  7. 2026Reconstructing Humans and Objects in Interaction using Large Reconstruction Modelscited by 0
  8. 2026GraspHOI: Full-Body 3D Human-Object Reconstruction with Finger-Level Grasps from a Single In-the-Wild Imagecited by 0