mlsanityData to 13 Oct 2026

Reconstructing Hands in 3D with Transformers

arXiv 2312.05251 ↗First posted 8 Dec 2023cs.CVcited by 300

Not on the map. Why?

Cited by 32 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present an approach that can reconstruct hands in 3D from monocular input. Our approach for Hand Mesh Recovery, HaMeR, follows a fully transformer-based architecture and can analyze hands with significantly increased accuracy and robustness compared to previous work. The key to HaMeR's success lies in scaling up both the data used for training and the capacity of the deep network for hand reconstruction. For training data, we combine multiple datasets that contain 2D or 3D hand annotations. For the deep model, we use a large scale Vision Transformer architecture. Our final model consistently outperforms the previous baselines on popular 3D hand pose benchmarks. To further evaluate the effect of our design in non-controlled settings, we annotate existing in-the-wild datasets with 2D hand keypoint annotations. On this newly collected dataset of annotations, HInt, we demonstrate significant improvements over existing baselines. We make our code, data and models available on the project website: https://geopavlakos.github.io/hamer/.

Key facts. Cited by 300 papers in the map (data through 13 Oct 2026). Builds on 55 of its 65 references that are in the map. First posted on arXiv 8 Dec 2023.

Builds on

55 of 65 references.

References · 55

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  3. 2020Language Models are Few-Shot Learnerscited by 32,320
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,777
  6. 2023GPT-4 Technical Reportcited by 18,278
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,550
  8. 2023Segment Anythingcited by 8,195
  9. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,535
  10. 2017End-to-end Recovery of Human Shape and Posecited by 883
  11. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 792
  12. 2019Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loopcited by 625
38 more
  1. 2022ViTPose: Simple Vision Transformer Baselines for Human Pose Estimationcited by 452
  2. 2020End-to-End Human Pose and Mesh Reconstruction with Transformerscited by 417
  3. 2019Learning joint reconstruction of hands and manipulated objectscited by 377
  4. 2023Humans in 4D: Reconstructing and Tracking Humans with Transformerscited by 345
  5. 2021DexYCB: A Benchmark for Capturing Hand Grasping of Objectscited by 322
  6. 2022Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activitiescited by 285
  7. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  8. 2020I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Imagecited by 274
  9. 2019FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Imagescited by 271
  10. 2021Mesh Graphormercited by 264
  11. 2020Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Posecited by 248
  12. 2020InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Imagecited by 245
  13. 2017Learning to Detect Human-Object Interactionscited by 238
  14. 2017Learning to Estimate 3D Hand Pose from Single RGB Imagescited by 234
  15. 2017Hand Keypoint Detection in Single Images using Multiview Bootstrappingcited by 231
  16. 2022AlphaPose: Whole-Body Regional Multi-Person Pose Estimation and Tracking in Real-Timecited by 199
  17. 20193D Hand Shape and Pose from Images in the Wildcited by 190
  18. 2022Embodied Hands: Modeling and Capturing Hands and Bodies Togethercited by 180
  19. 2020Whole-Body Human Pose Estimation in the Wildcited by 178
  20. 20193D Hand Shape and Pose Estimation from a Single RGB Imagecited by 174
  21. 2021FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integrationcited by 168
  22. 2018Monocular Total Capture: Posing Face, Body, and Hands in the Wildcited by 162
  23. 2020Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wildcited by 146
  24. 2021Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Timecited by 140
  25. 2022EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relationscited by 140
  26. 2020Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstructioncited by 139
  27. 2019End-to-end Hand Mesh Recovery from a Monocular RGB Imagecited by 124
  28. 2021Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose Estimationcited by 123
  29. 2019Pushing the Envelope for RGB-based Dense 3D Hand Pose Estimation via Neural Renderingcited by 115
  30. 2022HandOccNet: Occlusion-Robust 3D Hand Mesh Estimation Networkcited by 95
  31. 2022Interacting Attention Graph for Single Image Two-Hand Reconstructioncited by 87
  32. 2021ArtiBoost: Boosting Articulated 3D Hand-Object Pose Estimation via Online Exploration and Synthesiscited by 76
  33. 2021MobRecon: Mobile-Friendly Hand Mesh Reconstruction from Monocular Imagecited by 74
  34. 2022Collaborative Learning for Hand and Object Reconstruction with Attention-guided Graph Convolutioncited by 54
  35. 2021I2UV-HandNet: Image-to-UV Prediction Network for Accurate and High-fidelity 3D Hand Mesh Modelingcited by 52
  36. 2021Towards Accurate Alignment in Real-time 3D Hand-Mesh Reconstructioncited by 44
  37. 2023ACR: Attention Collaboration-based Regressor for Arbitrary Two-Hand Reconstructioncited by 37
  38. 20223D Interacting Hand Pose Estimation by Hand De-occlusion and Removalcited by 36

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 2; 2024: 44; 2025: 121; 2026: 133.

300 papers.

Sort:

By area:

  1. 2025EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Videocited by 198
  2. 2024HumanPlus: Humanoid Shadowing and Imitation from Humanscited by 166
  3. 2024DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulationcited by 149
  4. 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 125
  5. 2024WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wildcited by 121
  6. 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videoscited by 97
  7. 2024DINO-X: A Unified Vision Model for Open-World Object Detection and Understandingcited by 87
  8. 2025Uni3C: Unifying Precisely 3D-Enhanced Camera and Human Motion Controls for Video Generationcited by 75
  9. 2024EquiBot: SIM(3)-Equivariant Diffusion Policy for Generalizable and Data Efficient Learningcited by 73
  10. 2024Unpacking Failure Modes of Generative Policies: Runtime Monitoring of Consistency and Progresscited by 67
  11. 2025Large Video Planner Enables Generalizable Robot Controlcited by 67
  12. 2026SAM 3D Body: Robust Full-Body Human Mesh Recoverycited by 65
38 more
  1. 2024OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitationcited by 63
  2. 2025HaWoR: World-Space Hand Motion Reconstruction from Egocentric Videoscited by 62
  3. 2024DiffH2O: Diffusion-Based Synthesis of Hand-Object Interactions from Textual Descriptionscited by 61
  4. 2025Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learningcited by 58
  5. 2025ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learningcited by 57
  6. 2024Estimating Body and Hand Motion in an Ego-sensed Worldcited by 56
  7. 2024ParaHome: Parameterizing Everyday Home Activities Towards 3D Generative Modeling of Human-Object Interactionscited by 55
  8. 2024IDOL: Instant Photorealistic 3D Human Creation from a Single Imagecited by 55
  9. 2024Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstructioncited by 52
  10. 2024Dyn-HaMR: Recovering 4D Interacting Hand Motion from a Dynamic Cameracited by 50
  11. 2024Vision-based Manipulation from Single Human Video with Open-World Object Graphscited by 47
  12. 2024Hamba: Single-view 3D Hand Reconstruction with Graph-guided Bi-Scanning Mambacited by 45
  13. 2024Hand-Object Interaction Pretraining from Videoscited by 38
  14. 2024RealisDance: Equip controllable character animation with realistic handscited by 37
  15. 2024R+X: Retrieval and Execution from Everyday Human Videoscited by 36
  16. 2025DreamActor-M1: Holistic, Expressive and Robust Human Image Animation with Hybrid Guidancecited by 36
  17. 2024EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wildcited by 34
  18. 2025DexWild: Dexterous Human Interactions for In-the-Wild Robot Policiescited by 34
  19. 20233D Hand Pose Estimation in Everyday Egocentric Imagescited by 33
  20. 2024GigaHands: A Massive Annotated Dataset of Bimanual Hand Activitiescited by 33
  21. 2025SMPLest-X: Ultimate Scaling for Expressive Human Pose and Shape Estimationcited by 31
  22. 2025ZeroMimic: Distilling Robotic Manipulation Skills from Web Videoscited by 31
  23. 2025EgoZero: Robot Learning from Smart Glassescited by 31
  24. 2025World Models for Learning Dexterous Hand-Object Interactions from Human Videoscited by 30
  25. 2025RealisDance-DiT: Simple yet Strong Baseline towards Controllable Character Animation in the Wildcited by 29
  26. 2025Crossing the Human-Robot Embodiment Gap with Sim-to-Real RL using One Human Demonstrationcited by 28
  27. 2025SPIDER: Scalable Physics-Informed Dexterous Retargetingcited by 28
  28. 2024HO-Cap: A Capture System and Dataset for 3D Reconstruction and Pose Tracking of Hand-Object Interactioncited by 26
  29. 2025X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Realcited by 25
  30. 2024Reconstructing Hand-Held Objects in 3D from Images and Videoscited by 24
  31. 2024Bimanual Dexterity for Complex Taskscited by 23
  32. 2025Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flowcited by 23
  33. 2025Reconstructing Humans with a Biomechanically Accurate Skeletoncited by 22
  34. 2025GUAVA: Generalizable Upper Body 3D Gaussian Avatarcited by 22
  35. 2024Expressive Gaussian Human Avatars from Monocular RGB Videocited by 21
  36. 2025DreamActor-H1: High-Fidelity Human-Product Demonstration Video Generation via Motion-designed Diffusion Transformerscited by 21
  37. 2024AniMer: Animal Pose and Shape Estimation Using Family Aware Transformercited by 20
  38. 2025Challenges and Trends in Egocentric Vision: A Surveycited by 20

50 most cited of 300 All 300 (JSON)