# EgoTouch: On-Body Touch Input Using AR/VR Headset Cameras

arXiv: https://arxiv.org/abs/2509.01786 · HTML: https://mlsanity.com/paper/2509.01786/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 45 references that are in the map.
- First posted on arXiv 1 Sep 2025.

## Builds on (most cited first, 4 of 4)

- [FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterization](https://mlsanity.com/paper/2303.14189/) (2023)
- [MediaPipe Hands: On-device Real-time Hand Tracking](https://mlsanity.com/paper/2006.10214/) (2020)
- [PressureVision: Estimating Hand Pressure from a Single RGB Image](https://mlsanity.com/paper/2203.10385/) (2022)
- [PressureVision++: Estimating Fingertip Pressure from Diverse RGB Images](https://mlsanity.com/paper/2301.02310/) (2023)

## Built on by (most cited first, 5 of 5)

- [SurfaceXR: Fusing Smartwatch IMUs and Egocentric Hand Pose for Seamless Surface Interactions](https://mlsanity.com/paper/2603.19529/) (2026)
- [DeltaDorsal: Enhancing Hand Pose Estimation with Dorsal Features in Egocentric Views](https://mlsanity.com/paper/2601.15516/) (2026)
- [EclipseTouch: Touch Segmentation on Ad Hoc Surfaces using Worn Infrared Shadow Casting](https://mlsanity.com/paper/2509.03430/) (2025)
- [Contact-Aware Refinement of Human Pose Pseudo-Ground Truth via Bioimpedance Sensing](https://mlsanity.com/paper/2512.04862/) (2025)
- [Empirical Evaluation of Multi-Modal Touch Detection in Over-the-Shoulder Video Surveillance](https://mlsanity.com/paper/2606.29504/) (2026)

## Abstract (arXiv)

In augmented and virtual reality (AR/VR) experiences, a user's arms and hands can provide a convenient and tactile surface for touch input. Prior work has shown on-body input to have significant speed, accuracy, and ergonomic benefits over in-air interfaces, which are common today. In this work, we demonstrate high accuracy, bare hands (i.e., no special instrumentation of the user) skin input using just an RGB camera, like those already integrated into all modern XR headsets. Our results show this approach can be accurate, and robust across diverse lighting conditions, skin tones, and body motion (e.g., input while walking). Finally, our pipeline also provides rich input metadata including touch force, finger identification, angle of attack, and rotation. We believe these are the requisite technical ingredients to more fully unlock on-skin interfaces that have been well motivated in the HCI literature but have lacked robust and practical methods.