# EVFormer: An Egocentric Vision-EMG Bidirectional Attention Model for Bimanual Hand Pose Estimation

arXiv: https://arxiv.org/abs/2610.06970 · HTML: https://mlsanity.com/paper/2610.06970/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 40 references that are in the map.
- First posted on arXiv 3 Oct 2026.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Reconstructing Hands in 3D with Transformers](https://mlsanity.com/paper/2312.05251/) (2023)
- [H2O: Two Hands Manipulating Objects for First Person Interaction Recognition](https://mlsanity.com/paper/2104.11181/) (2021)
- [InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/2008.09309/) (2020)
- [HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos](https://mlsanity.com/paper/2411.19167/) (2024)
- [WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wild](https://mlsanity.com/paper/2409.12259/) (2024)
- [UmeTrack: Unified multi-view end-to-end hand tracking for VR](https://mlsanity.com/paper/2211.00099/) (2022)
- [emg2pose: A Large and Diverse Benchmark for Surface Electromyographic Hand Pose Estimation](https://mlsanity.com/paper/2412.02725/) (2024)
- [Multimodal Fusion of EMG and Vision for Human Grasp Intent Inference in Prosthetic Hand Control](https://mlsanity.com/paper/2104.03893/) (2021)
- [EMGBench: Benchmarking Out-of-Distribution Generalization and Adaptation for Electromyography](https://mlsanity.com/paper/2410.23625/) (2024)
- [TinyMyo: a Tiny Foundation Model for Flexible EMG Signal Processing at the Edge](https://mlsanity.com/paper/2512.15729/) (2025)
- [WristPP: A Wrist-Worn System for Hand Pose And Pressure Estimation](https://mlsanity.com/paper/2603.00606/) (2026)
- [Scaling and Distilling Transformer Models for sEMG](https://mlsanity.com/paper/2507.22094/) (2025)
- [EgoEMG: A Multimodal Egocentric Dataset with Bilateral EMG and Vision for Hand Pose Estimation](https://mlsanity.com/paper/2605.05712/) (2026)
- [emg2tendon: From sEMG Signals to Tendon Control in Musculoskeletal Hands](https://mlsanity.com/paper/2508.08269/) (2025)
- [DexEMG: Towards Dexterous Teleoperation System via EMG2Pose Generalization](https://mlsanity.com/paper/2603.05861/) (2026)

## Abstract (arXiv)

Egocentric bimanual hand pose estimation is important for virtual interaction, wearable control, and rehabilitation, but visual observations are often degraded by self-occlusion, hand-hand contact, and object manipulation. We propose EVFormer, a multimodal framework that combines the current RGB frame with the preceding 200 ms of bilateral wrist surface electromyography (sEMG) to estimate 44 finger and wrist joint angles. EVFormer separately encodes visual spatial features and sEMG temporal features, enables cross-modal information exchange through sequential bidirectional cross-attention, and integrates the two modalities using feature-wise gated fusion. We evaluate EVFormer in a single-participant feasibility study using one synchronized public EgoEMG recording with chronologically separated training, validation, and test splits. On 296 test samples, EVFormer achieves a mean absolute error of 11.482 degrees, compared with 13.228-13.610 degrees for vision-only, sEMG-only, late-fusion, and training-mean baselines. This corresponds to relative error reductions of 13.20% compared with the vision-only model and 14.23% compared with late fusion. EVFormer also achieves the lowest error in four of the five evaluated gesture classes. These results provide preliminary evidence that feature-level interaction between egocentric vision and sEMG can improve bimanual hand pose estimation. Further evaluation across participants, recording sessions, sensor placements, and real-world interaction conditions is required to establish the generalizability of the approach.