mlsanityData to 7 Oct 2026

Complementing Event Streams and RGB Frames for Hand Mesh Reconstruction

arXiv 2403.07346 ↗First posted 12 Mar 2024cs.CVcited by 14

Not on the map. Why?

Abstract (arXiv)

Reliable hand mesh reconstruction (HMR) from commonly-used color and depth sensors is challenging especially under scenarios with varied illuminations and fast motions. Event camera is a highly promising alternative for its high dynamic range and dense temporal resolution properties, but it lacks key texture appearance for hand mesh reconstruction. In this paper, we propose EvRGBHand -- the first approach for 3D hand mesh reconstruction with an event camera and an RGB camera compensating for each other. By fusing two modalities of data across time, space, and information dimensions,EvRGBHand can tackle overexposure and motion blur issues in RGB-based HMR and foreground scarcity and background overflow issues in event-based HMR. We further propose EvRGBDegrader, which allows our model to generalize effectively in challenging scenes, even when trained solely on standard scenes, thus reducing data acquisition costs. Experiments on real-world data demonstrate that EvRGBHand can effectively solve the challenging issues when using either type of camera alone via retaining the merits of both, and shows the potential of generalization to outdoor scenes and another type of event camera.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 45 of its 72 references that are in the map. First posted on arXiv 12 Mar 2024.

Builds on

45 of 72 references.

References · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  5. 2018CBAM: Convolutional Block Attention Modulecited by 2,463
  6. 2017Deformable Convolutional Networkscited by 1,481
  7. 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,428
  8. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,417
  9. 2018Self-Attention with Relative Position Representationscited by 1,112
  10. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  11. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  12. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
33 more
  1. 2020End-to-End Human Pose and Mesh Reconstruction with Transformerscited by 417
  2. 2018Unsupervised Event-based Learning of Optical Flow, Depth, and Egomotioncited by 304
  3. 2019FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Imagescited by 271
  4. 2021Mesh Graphormercited by 264
  5. 2020InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Imagecited by 244
  6. 2017Hand Keypoint Detection in Single Images using Multiview Bootstrappingcited by 231
  7. 20193D Hand Shape and Pose from Images in the Wildcited by 190
  8. 2020MediaPipe Hands: On-device Real-time Hand Trackingcited by 157
  9. 2019Cost Volume Pyramid Based Depth Inference for Multi-View Stereocited by 154
  10. 2020Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wildcited by 146
  11. 2018Event-based Moving Object Detection and Trackingcited by 126
  12. 2019End-to-end Hand Mesh Recovery from a Monocular RGB Imagecited by 124
  13. 2020Epipolar Transformerscited by 117
  14. 2021Event-Based Fusion for Motion Deblurring with Cross-modal Attentioncited by 112
  15. 2022Time Lens++: Event-based Frame Interpolation with Parametric Non-linear Flow and Multi-scale Fusioncited by 110
  16. 2020Monocular Real-time Hand Shape and Motion Capture using Multi-modal Datacited by 106
  17. 2022Interacting Attention Graph for Single Image Two-Hand Reconstructioncited by 87
  18. 2022Cross-Attention of Disentangled Modalities for 3D Human Mesh Recovery with Transformerscited by 81
  19. 2021MobRecon: Mobile-Friendly Hand Mesh Reconstruction from Monocular Imagecited by 74
  20. 2019EventCap: Monocular 3D Capture of High-Speed Human Motions using an Event Cameracited by 65
  21. 2021Model-based 3D Hand Reconstruction via Self-Supervised Learningcited by 65
  22. 2020Parameter Efficient Multimodal Transformers for Video Representation Learningcited by 52
  23. 2020EventHands: Real-Time Neural 3D Hand Pose Estimation from an Event Streamcited by 48
  24. 2023Frame-Event Alignment and Fusion Network for High Frame Rate Trackingcited by 43
  25. 2021EventHPE: Event-based 3D Human Pose and Shape Estimationcited by 42
  26. 2022Multi-Bracket High Dynamic Range Imaging with Event Camerascited by 37
  27. 2020SeqHAND:RGB-Sequence-Based 3D Hand Pose and Shape Estimationcited by 34
  28. 2023Learning Spatial-Temporal Implicit Neural Representations for Event-Guided Video Super-Resolutioncited by 33
  29. 2021Differentiable Event Stream Simulator for Non-Rigid 3D Trackingcited by 23
  30. 2020JGR-P2O: Joint Graph Reasoning based Pixel-to-Offset Prediction Network for 3D Hand Pose Estimation from a Single Depth Imagecited by 20
  31. 2018A Mixed Classification-Regression Framework for 3D Pose Estimation from 2D Imagescited by 19
  32. 2021HandVoxNet++: 3D Hand Shape and Pose Estimation using Voxel-Based Neural Networkscited by 11
  33. 2023POEM: Reconstructing Hand in a Point Embedded Multi-view Stereocited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 7; 2026: 3.

14 papers.

Sort:

  1. 2024Recent Event Camera Innovations: A Surveycited by 67
  2. 2025EventEgo3D++: 3D Human Motion Capture from a Head-Mounted Event Cameracited by 6
  3. 2025Beyond conventional vision: RGB-event fusion for robust object detection in dynamic traffic scenarioscited by 6
  4. 2025Efficient Event-Based Semantic Segmentation via Exploiting Frame-Event Fusion: A Hybrid Neural Network Approachcited by 5
  5. 2024Dynamic EventNeRF: Reconstructing General Dynamic Scenes from Multi-view RGB and Event Streamscited by 3
  6. 2025EgoEvGesture: Gesture Recognition Based on Egocentric Event Cameracited by 3
  7. 2025From Sharp to Blur: Unsupervised Domain Adaptation for 2D Human Pose Estimation Under Extreme Motion Blur Using Event Camerascited by 3
  8. 2025Event Camera Guided Visual Media Restoration & 3D Reconstruction: A Surveycited by 3
  9. 2025Leveraging RGB Images for Pre-Training of Event-Based Hand Pose Estimationcited by 3
  10. 20243D Human Scan With A Moving Event Cameracited by 2
  11. 2024Steering Prediction via a Multi-Sensor System for Autonomous Racingcited by 2
  12. 2026EgoEV-HandPose: Egocentric 3D Hand Pose Estimation and Gesture Recognition with Stereo Event Camerascited by 2
2 more
  1. 2026GeoHand: Unlocking Prior Geometry Knowledge for Monocular 3D Hand Reconstructioncited by 1
  2. 2026EventEgoHands++: Event-based Egocentric 3D Hand Mesh Reconstruction with Real Datasetcited by 0