mlsanityData to 14 Oct 2026

Multi-view Hand Reconstruction with a Point-Embedded Transformer

arXiv 2408.10581 ↗First posted 20 Aug 2024cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

This work introduces a novel and generalizable multi-view Hand Mesh Reconstruction (HMR) model, named POEM, designed for practical use in real-world hand motion capture scenarios. The advances of the POEM model consist of two main aspects. First, concerning the modeling of the problem, we propose embedding a static basis point within the multi-view stereo space. A point represents a natural form of 3D information and serves as an ideal medium for fusing features across different views, given its varied projections across these views. Consequently, our method harnesses a simple yet effective idea: a complex 3D hand mesh can be represented by a set of 3D basis points that 1) are embedded in the multi-view stereo, 2) carry features from the multi-view images, and 3) encompass the hand in it. The second advance lies in the training strategy. We utilize a combination of five large-scale multi-view datasets and employ randomization in the number, order, and poses of the cameras. By processing such a vast amount of data and a diverse array of camera configurations, our model demonstrates notable generalizability in the real-world applications. As a result, POEM presents a highly practical, plug-and-play solution that enables user-friendly, cost-effective multi-view motion capture for both left and right hands. The model and source codes are available at https://github.com/JubSteven/POEM-v2.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 52 of its 67 references that are in the map. First posted on arXiv 20 Aug 2024.

Builds on

52 of 67 references.

References · 52

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  3. 2020End-to-End Object Detection with Transformerscited by 7,414
  4. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  5. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  6. 2020Point Transformercited by 1,470
  7. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,420
  8. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,236
  9. 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,205
  10. 2018MVSNet: Depth Inference for Unstructured Multi-view Stereocited by 788
  11. 2018PointCNN: Convolution On X-Transformed Pointscited by 764
  12. 2020PCT: Point cloud transformercited by 732
38 more
  1. 2018Pixel2Mesh: Generating 3D Mesh Models from Single RGB Imagescited by 651
  2. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 552
  3. 2020End-to-End Human Pose and Mesh Reconstruction with Transformerscited by 418
  4. 2019Learning joint reconstruction of hands and manipulated objectscited by 378
  5. 2021DexYCB: A Benchmark for Capturing Hand Grasping of Objectscited by 325
  6. 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 312
  7. 2023Reconstructing Hands in 3D with Transformerscited by 306
  8. 2020I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Imagecited by 274
  9. 2019FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Imagescited by 271
  10. 2021Mesh Graphormercited by 265
  11. 2020Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Posecited by 248
  12. 2020InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Imagecited by 245
  13. 2017Learning a Multi-View Stereo Machinecited by 194
  14. 2020Grasping Field: Learning Implicit Representations for Human Graspscited by 179
  15. 20193D Hand Shape and Pose Estimation from a Single RGB Imagecited by 174
  16. 2020MediaPipe Hands: On-device Real-time Hand Trackingcited by 161
  17. 2023AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation Systemcited by 152
  18. 2019Point-Based Multi-View Stereo Networkcited by 147
  19. 2021Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Timecited by 141
  20. 2022OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interactioncited by 131
  21. 2017SurfaceNet: An End-to-end 3D Neural Network for Multiview Stereopsiscited by 128
  22. 2021Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose Estimationcited by 123
  23. 2020Epipolar Transformerscited by 117
  24. 2019Efficient Learning on Point Clouds with Basis Point Setscited by 103
  25. 2022HandOccNet: Occlusion-Robust 3D Hand Mesh Estimation Networkcited by 96
  26. 2021ArtiBoost: Boosting Articulated 3D Hand-Object Pose Estimation via Online Exploration and Synthesiscited by 76
  27. 2021MobRecon: Mobile-Friendly Hand Mesh Reconstruction from Monocular Imagecited by 74
  28. 2021Model-based 3D Hand Reconstruction via Self-Supervised Learningcited by 65
  29. 2021Direct Multi-view Multi-person 3D Pose Estimationcited by 64
  30. 2021Camera-Space Hand Mesh Recovery via Semantic Aggregation and Adaptive 2D-1D Registrationcited by 59
  31. 2020Lightweight Multi-View 3D Pose Estimation through Camera-Disentangled Representationcited by 57
  32. 2021I2UV-HandNet: Image-to-UV Prediction Network for Accurate and High-fidelity 3D Hand Mesh Modelingcited by 52
  33. 2022UmeTrack: Unified multi-view end-to-end hand tracking for VRcited by 50
  34. 2024Hamba: Single-view 3D Hand Reconstruction with Graph-guided Bi-Scanning Mambacited by 45
  35. 2017Interpretable Transformations with Encoder-Decoder Networkscited by 34
  36. 2023A Probabilistic Attention Model with Occlusion-aware Texture Regression for 3D Hand Reconstruction from a Single RGB Imagecited by 32
  37. 2021HO-3D_v3: Improving the Accuracy of Hand-Object Annotations of the HO-3D Datasetcited by 26
  38. 2024HandBooster: Boosting 3D Hand-Mesh Reconstruction by Conditional Synthesis and Sampling of Hand-Object Interactionscited by 13

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Can LLMs See Without Pixels? Benchmarking Spatial Intelligence from Textual Descriptionscited by 7
  2. 2026HGGT: Robust and Flexible 3D Hand Mesh Reconstruction from Uncalibrated Imagescited by 3
  3. 2026LIDEA: Human-to-Robot Imitation Learning via Implicit Feature Distillation and Explicit Geometry Alignmentcited by 1
  4. 2026High-Fidelity 4D Hand-Object Capture via Multi-View Spatiotemporal Tracking and Physics-Aware Gaussianscited by 0
  5. 2026MHE-Former: Multi-Hypothesis Transformers via Entropy Maximization for 3D Mesh Recoverycited by 0
  6. 2026ESTHER: Egocentric Stereo Hand Estimation and Reconstruction in the Wildcited by 0