SurfaceXR: Fusing Smartwatch IMUs and Egocentric Hand Pose for Seamless Surface Interactions
Not on the map. Why?
Abstract (arXiv)
Mid-air gestures in Extended Reality (XR) often cause fatigue and imprecision. Surface-based interactions offer improved accuracy and comfort, but current egocentric vision methods struggle due to hand tracking challenges and unreliable surface plane estimation. We introduce SurfaceXR, a sensor fusion approach combining headset-based hand tracking with smartwatch IMU data to enable robust inputs on everyday surfaces. Our insight is that these modalities are complementary: hand tracking provides 3D positional data while IMUs capture high-frequency motion. A 21-participant study validates SurfaceXR's effectiveness for touch tracking and 8-class gesture recognition, demonstrating significant improvements over single-modality approaches.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 9 of its 54 references that are in the map. First posted on arXiv 19 Mar 2026.
Builds on
9 of 54 references.
References · 9
- 2016Temporal Convolutional Networks for Action Segmentation and Detectioncited by 504
- 2017Gated Multimodal Units for Information Fusioncited by 244
- 2022Enabling hand gesture customization on wrist-worn devicescited by 15
- 2023PressureVision++: Estimating Fingertip Pressure from Diverse RGB Imagescited by 12
- 2025EgoTouch: On-Body Touch Input Using AR/VR Headset Camerascited by 5
- 2024TouchInsight: Uncertainty-aware Rapid Touch and Text Input for Mixed Reality from Egocentric Visioncited by 4
- 2024TapType: Ten-finger text entry on everyday surfaces via Bayesian inferencecited by 4
- 2025HaloTouch: Using IR Multi-path Interference to Support Touch Interactions With General Surfacescited by 2
- 2024Hovering Over the Key to Text Input in XRcited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.