# SurfaceXR: Fusing Smartwatch IMUs and Egocentric Hand Pose for Seamless Surface Interactions

arXiv: https://arxiv.org/abs/2603.19529 · HTML: https://mlsanity.com/paper/2603.19529/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 54 references that are in the map.
- First posted on arXiv 19 Mar 2026.

## Builds on (most cited first, 9 of 9)

- [Temporal Convolutional Networks for Action Segmentation and Detection](https://mlsanity.com/paper/1611.05267/) (2016)
- [Gated Multimodal Units for Information Fusion](https://mlsanity.com/paper/1702.01992/) (2017)
- [Enabling hand gesture customization on wrist-worn devices](https://mlsanity.com/paper/2203.15239/) (2022)
- [PressureVision++: Estimating Fingertip Pressure from Diverse RGB Images](https://mlsanity.com/paper/2301.02310/) (2023)
- [EgoTouch: On-Body Touch Input Using AR/VR Headset Cameras](https://mlsanity.com/paper/2509.01786/) (2025)
- [TouchInsight: Uncertainty-aware Rapid Touch and Text Input for Mixed Reality from Egocentric Vision](https://mlsanity.com/paper/2410.05940/) (2024)
- [TapType: Ten-finger text entry on everyday surfaces via Bayesian inference](https://mlsanity.com/paper/2410.06001/) (2024)
- [HaloTouch: Using IR Multi-path Interference to Support Touch Interactions With General Surfaces](https://mlsanity.com/paper/2503.01197/) (2025)
- [Hovering Over the Key to Text Input in XR](https://mlsanity.com/paper/2406.09579/) (2024)

## Built on by (most cited first, 2 of 2)

- [Empirical Evaluation of Multi-Modal Touch Detection in Over-the-Shoulder Video Surveillance](https://mlsanity.com/paper/2606.29504/) (2026)
- [EgoWholeView: Camera-Only Head-Mounted Capture of Body and Bimanual Hand Motion for Embodied Demonstrations](https://mlsanity.com/paper/2610.16256/) (2026)

## Abstract (arXiv)

Mid-air gestures in Extended Reality (XR) often cause fatigue and imprecision. Surface-based interactions offer improved accuracy and comfort, but current egocentric vision methods struggle due to hand tracking challenges and unreliable surface plane estimation. We introduce SurfaceXR, a sensor fusion approach combining headset-based hand tracking with smartwatch IMU data to enable robust inputs on everyday surfaces. Our insight is that these modalities are complementary: hand tracking provides 3D positional data while IMUs capture high-frequency motion. A 21-participant study validates SurfaceXR's effectiveness for touch tracking and 8-class gesture recognition, demonstrating significant improvements over single-modality approaches.