mlsanityData to 14 Oct 2026

HandOS: 3D Hand Reconstruction in One Stage

arXiv 2412.01537 ↗First posted 2 Dec 2024cs.CV cs.GRcited by 9

Not on the map. Why?

Abstract (arXiv)

Existing approaches of hand reconstruction predominantly adhere to a multi-stage framework, encompassing detection, left-right classification, and pose estimation. This paradigm induces redundant computation and cumulative errors. In this work, we propose HandOS, an end-to-end framework for 3D hand reconstruction. Our central motivation lies in leveraging a frozen detector as the foundation while incorporating auxiliary modules for 2D and 3D keypoint estimation. In this manner, we integrate the pose estimation capacity into the detection framework, while at the same time obviating the necessity of using the left-right category as a prerequisite. Specifically, we propose an interactive 2D-3D decoder, where 2D joint semantics is derived from detection cues while 3D representation is lifted from those of 2D joints. Furthermore, hierarchical attention is designed to enable the concurrent modeling of 2D joints, 3D vertices, and camera translation. Consequently, we achieve an end-to-end integration of hand detection, 2D pose estimation, and 3D mesh reconstruction within a one-stage framework, so that the above multi-stage drawbacks are overcome. Meanwhile, the HandOS reaches state-of-the-art performances on public benchmarks, e.g., 5.0 PA-MPJPE on FreiHand and 64.6% [email protected] on HInt-Ego4D. Project page: idea-research.github.io/HandOSweb.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 64 of its 83 references that are in the map. First posted on arXiv 2 Dec 2024.

Builds on

64 of 83 references.

References · 64

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2020End-to-End Object Detection with Transformerscited by 7,414
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  6. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  7. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  8. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,612
  9. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,319
  10. 2021YOLOX: Exceeding YOLO Series in 2021cited by 1,071
  11. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  12. 2022ViTPose: Simple Vision Transformer Baselines for Human Pose Estimationcited by 456
38 more
  1. 2020End-to-End Human Pose and Mesh Reconstruction with Transformerscited by 418
  2. 2019Learning joint reconstruction of hands and manipulated objectscited by 378
  3. 2021DexYCB: A Benchmark for Capturing Hand Grasping of Objectscited by 325
  4. 2023Reconstructing Hands in 3D with Transformerscited by 306
  5. 2020I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Imagecited by 274
  6. 2019FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Imagescited by 271
  7. 2021Mesh Graphormercited by 265
  8. 2020InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Imagecited by 245
  9. 20193D Hand Shape and Pose from Images in the Wildcited by 190
  10. 2020Whole-Body Human Pose Estimation in the Wildcited by 178
  11. 2021OTA: Optimal Transport Assignment for Object Detectioncited by 178
  12. 20193D Hand Shape and Pose Estimation from a Single RGB Imagecited by 174
  13. 2021FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integrationcited by 169
  14. 2023FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterizationcited by 162
  15. 2021Hand-Object Contact Consistency Reasoning for Human Grasps Generationcited by 160
  16. 2020Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wildcited by 146
  17. 2020Reconstructing Hand-Object Interactions in the Wildcited by 144
  18. 2021Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Timecited by 141
  19. 2022DexGraspNet: A Large-Scale Robotic Dexterous Grasp Dataset for General Objects Based on Simulationcited by 137
  20. 2020CPF: Learning a Contact Potential Field to Model the Hand-Object Interactioncited by 128
  21. 2019End-to-end Hand Mesh Recovery from a Monocular RGB Imagecited by 124
  22. 2021Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose Estimationcited by 123
  23. 2024Grounding DINO 1.5: Advance the "Edge" of Open-Set Object Detectioncited by 122
  24. 2022Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learningcited by 119
  25. 2019Pushing the Envelope for RGB-based Dense 3D Hand Pose Estimation via Neural Renderingcited by 115
  26. 2020HOPE-Net: A Graph-based Model for Hand-Object Pose Estimationcited by 112
  27. 2018Hand Pose Estimation via Latent 2.5D Heatmap Regressioncited by 111
  28. 2020Monocular Real-time Hand Shape and Motion Capture using Multi-modal Datacited by 106
  29. 2020Weakly Supervised 3D Hand Pose Estimation via Biomechanical Constraintscited by 97
  30. 2022HandOccNet: Occlusion-Robust 3D Hand Mesh Estimation Networkcited by 96
  31. 2022Interacting Attention Graph for Single Image Two-Hand Reconstructioncited by 87
  32. 2022Cross-Attention of Disentangled Modalities for 3D Human Mesh Recovery with Transformerscited by 81
  33. 2021ArtiBoost: Boosting Articulated 3D Hand-Object Pose Estimation via Online Exploration and Synthesiscited by 76
  34. 2021MobRecon: Mobile-Friendly Hand Mesh Reconstruction from Monocular Imagecited by 74
  35. 2022YOLO-Pose: Enhancing YOLO for Multi Person Pose Estimation Using Object Keypoint Similarity Losscited by 68
  36. 2021Model-based 3D Hand Reconstruction via Self-Supervised Learningcited by 65
  37. 2021Camera-Space Hand Mesh Recovery via Semantic Aggregation and Adaptive 2D-1D Registrationcited by 59
  38. 2019A2J: Anchor-to-Joint Regression Network for 3D Articulated Pose Estimation from a Single Depth Imagecited by 56

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 6.

9 papers.

  1. 2026ForeHOI: Feed-forward 3D Object Reconstruction from Daily Hand-Object Interaction Videoscited by 7
  2. 2025Efficient and Scalable Monocular Human-Object Interaction Motion Reconstructioncited by 3
  3. 2026PAM: A Pose-Appearance-Motion Engine for Sim-to-Real HOI Video Generationcited by 3
  4. 2026GeoHand: Unlocking Prior Geometry Knowledge for Monocular 3D Hand Reconstructioncited by 2
  5. 2026Hand-Object Interaction in the Age of Large Foundation Models:Reconstruction, Generation, and Embodied Transfercited by 1
  6. 2025Enhancing Monocular 3D Hand Reconstruction with Learned Texture Priorscited by 0
  7. 2025Cascaded Diffusion Framework for Probabilistic Coarse-to-Fine Hand Pose Estimationcited by 0
  8. 2026ScaleHP: Scale-Mediated Optimization of Coupled Errors for Metric-Space Hand Pose Estimationcited by 0
  9. 2026MHE-Former: Multi-Hypothesis Transformers via Entropy Maximization for 3D Mesh Recoverycited by 0