# Force-Aware Interface via Electromyography for Natural VR/AR Interaction

arXiv: https://arxiv.org/abs/2210.01225 · HTML: https://mlsanity.com/paper/2210.01225/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 116 references that are in the map.
- First posted on arXiv 3 Oct 2022.

## Builds on (most cited first, 19 of 19)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies](https://mlsanity.com/paper/1801.01615/) (2018)
- [Stochastic Scene-Aware Motion Prediction](https://mlsanity.com/paper/2108.08284/) (2021)
- [3D Hand Shape and Pose from Images in the Wild](https://mlsanity.com/paper/1902.03451/) (2019)
- [Embodied Hands: Modeling and Capturing Hands and Bodies Together](https://mlsanity.com/paper/2201.02610/) (2022)
- [Pixel Codec Avatars](https://mlsanity.com/paper/2104.04638/) (2021)
- [Use the Force, Luke! Learning to Predict Physical Forces by Simulating Effects](https://mlsanity.com/paper/2003.12045/) (2020)
- [Dense 3D Regression for Hand Pose Estimation](https://mlsanity.com/paper/1711.08996/) (2017)
- [Expressive Telepresence via Modular Codec Avatars](https://mlsanity.com/paper/2008.11789/) (2020)
- [WiGest: A Ubiquitous WiFi-based Gesture Recognition System](https://mlsanity.com/paper/1501.04301/) (2015)
- [Discovering Diverse Athletic Jumping Strategies](https://mlsanity.com/paper/2105.00371/) (2021)
- [High-fidelity Face Tracking for AR/VR via Deep Lighting Adaptation](https://mlsanity.com/paper/2103.15876/) (2021)
- [Real-Time Tactile Grasp Force Sensing Using Fingernail Imaging via Deep Neural Networks](https://mlsanity.com/paper/2109.15231/) (2021)
- [Comparison of Constrained and Unconstrained Human Grasp Forces Using Fingernail Imaging and Visual Servoing](https://mlsanity.com/paper/2004.02311/) (2020)

## Built on by (most cited first, 6 of 6)

- [ClothCombo: Modeling Inter-Cloth Interaction for Draping Multi-Layered Clothes](https://mlsanity.com/paper/2304.03492/) (2023)
- [Posture-Informed Muscular Force Learning for Robust Hand Pressure Estimation](https://mlsanity.com/paper/2410.23629/) (2024)
- [Toward Optimized VR/AR Ergonomics: Modeling and Predicting User Neck Muscle Contraction](https://mlsanity.com/paper/2308.14841/) (2023)
- [Spatial Visibility and Temporal Dynamics: Revolutionizing Field of View Prediction in Adaptive Point Cloud Video Streaming](https://mlsanity.com/paper/2409.18236/) (2024)
- [IsoNet: Causal Analysis of Multimodal Transformers for Neuromuscular Gesture Classification](https://mlsanity.com/paper/2506.16744/) (2025)
- [LiteEMG-FM: An Efficient and Deployable Foundation Model for Robust EMG Sensing](https://mlsanity.com/paper/2610.02497/) (2026)

## Abstract (arXiv)

While tremendous advances in visual and auditory realism have been made for virtual and augmented reality (VR/AR), introducing a plausible sense of physicality into the virtual world remains challenging. Closing the gap between real-world physicality and immersive virtual experience requires a closed interaction loop: applying user-exerted physical forces to the virtual environment and generating haptic sensations back to the users. However, existing VR/AR solutions either completely ignore the force inputs from the users or rely on obtrusive sensing devices that compromise user experience. By identifying users' muscle activation patterns while engaging in VR/AR, we design a learning-based neural interface for natural and intuitive force inputs. Specifically, we show that lightweight electromyography sensors, resting non-invasively on users' forearm skin, inform and establish a robust understanding of their complex hand activities. Fuelled by a neural-network-based model, our interface can decode finger-wise forces in real-time with 3.3% mean error, and generalize to new users with little calibration. Through an interactive psychophysical study, we show that human perception of virtual objects' physical properties, such as stiffness, can be significantly enhanced by our interface. We further demonstrate that our interface enables ubiquitous control via finger tapping. Ultimately, we envision our findings to push forward research towards more realistic physicality in future VR/AR.