mlsanityData to 7 Oct 2026

POEM: Reconstructing Hand in a Point Embedded Multi-view Stereo

arXiv 2304.04038 ↗First posted 8 Apr 2023cs.CVcited by 9

Not on the map. Why?

Abstract (arXiv)

Enable neural networks to capture 3D geometrical-aware features is essential in multi-view based vision tasks. Previous methods usually encode the 3D information of multi-view stereo into the 2D features. In contrast, we present a novel method, named POEM, that directly operates on the 3D POints Embedded in the Multi-view stereo for reconstructing hand mesh in it. Point is a natural form of 3D information and an ideal medium for fusing features across views, as it has different projections on different views. Our method is thus in light of a simple yet effective idea, that a complex 3D hand mesh can be represented by a set of 3D points that 1) are embedded in the multi-view stereo, 2) carry features from the multi-view images, and 3) encircle the hand. To leverage the power of points, we design two operations: point-based feature fusion and cross-set point attention mechanism. Evaluation on three challenging multi-view datasets shows that POEM outperforms the state-of-the-art in hand mesh reconstruction. Code and models are available for research at https://github.com/lixiny/POEM.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 53 of its 59 references that are in the map. First posted on arXiv 8 Apr 2023.

Builds on

53 of 59 references.

References · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2020End-to-End Object Detection with Transformerscited by 7,384
  5. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  6. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  7. 2017Non-local Neural Networkscited by 2,290
  8. 2020Point Transformercited by 1,462
  9. 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,202
  10. 2018MVSNet: Depth Inference for Unstructured Multi-view Stereocited by 780
  11. 2018PointCNN: Convolution On X-Transformed Pointscited by 764
  12. 2020PCT: Point cloud transformercited by 730
38 more
  1. 2018PointConv: Deep Convolutional Networks on 3D Point Cloudscited by 656
  2. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 550
  3. 2020End-to-End Human Pose and Mesh Reconstruction with Transformerscited by 417
  4. 2019Learning joint reconstruction of hands and manipulated objectscited by 377
  5. 2016Convolutional Pose Machinescited by 325
  6. 2021DexYCB: A Benchmark for Capturing Hand Grasping of Objectscited by 320
  7. 2020Exploring Self-attention for Image Recognitioncited by 313
  8. 2020I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Imagecited by 274
  9. 2019FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Imagescited by 271
  10. 2019Convolutional Mesh Regression for Single-Image Human Shape Reconstructioncited by 270
  11. 2021Mesh Graphormercited by 264
  12. 2017Integral Human Pose Regressioncited by 255
  13. 2017Learning a Multi-View Stereo Machinecited by 194
  14. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videoscited by 194
  15. 20193D Hand Shape and Pose from Images in the Wildcited by 190
  16. 2020ContactPose: A Dataset of Grasps with Object Contact and Hand Posecited by 180
  17. 2022Embodied Hands: Modeling and Capturing Hands and Bodies Togethercited by 179
  18. 2020Grasping Field: Learning Implicit Representations for Human Graspscited by 178
  19. 20193D Hand Shape and Pose Estimation from a Single RGB Imagecited by 174
  20. 2019Learnable Triangulation of Human Posecited by 160
  21. 2019Point-Based Multi-View Stereo Networkcited by 147
  22. 2020Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wildcited by 146
  23. 2022OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interactioncited by 129
  24. 2017SurfaceNet: An End-to-end 3D Neural Network for Multiview Stereopsiscited by 128
  25. 2020CPF: Learning a Contact Potential Field to Model the Hand-Object Interactioncited by 127
  26. 2021Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose Estimationcited by 122
  27. 2020Epipolar Transformerscited by 117
  28. 2020Monocular Real-time Hand Shape and Motion Capture using Multi-modal Datacited by 106
  29. 2019Efficient Learning on Point Clouds with Basis Point Setscited by 101
  30. 2022HandOccNet: Occlusion-Robust 3D Hand Mesh Estimation Networkcited by 95
  31. 2019Cross View Fusion for 3D Human Pose Estimationcited by 90
  32. 2021Direct Multi-view Multi-person 3D Pose Estimationcited by 64
  33. 2021Camera-Space Hand Mesh Recovery via Semantic Aggregation and Adaptive 2D-1D Registrationcited by 59
  34. 2022AlignSDF: Pose-Aligned Signed Distance Fields for Hand-Object Reconstructioncited by 58
  35. 2020Lightweight Multi-View 3D Pose Estimation through Camera-Disentangled Representationcited by 57
  36. 2021I2UV-HandNet: Image-to-UV Prediction Network for Accurate and High-fidelity 3D Hand Mesh Modelingcited by 52
  37. 2017Harvesting Multiple Views for Marker-less 3D Human Pose Annotationscited by 48
  38. 2022UmeTrack: Unified multi-view end-to-end hand tracking for VRcited by 48

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 2; 2026: 3.

9 papers.

  1. 2024OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completioncited by 78
  2. 2025ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learningcited by 56
  3. 2024Complementing Event Streams and RGB Frames for Hand Mesh Reconstructioncited by 14
  4. 2024Multi-view Hand Reconstruction with a Point-Embedded Transformercited by 6
  5. 2024MLPHand: Real Time Multi-View 3D Hand Mesh Reconstruction via MLP Modelingcited by 3
  6. 2025Towards Dynamic 3D Reconstruction of Hand-Instrument Interaction in Ophthalmic Surgerycited by 2
  7. 2026EgoEV-HandPose: Egocentric 3D Hand Pose Estimation and Gesture Recognition with Stereo Event Camerascited by 2
  8. 2026UST-Hand: An Uncertainty-aware Spatiotemporal Point Cloud Interaction Network for 3D Self-supervised Hand Pose Estimationcited by 0
  9. 2026HandMvNet: Real-Time 3D Hand Pose Estimation Using Multi-View Cross-Attention Fusioncited by 0