# Integrated Sensing and Communications for Real-time Avatar Control in XR over 5G

arXiv: https://arxiv.org/abs/2606.23771 · HTML: https://mlsanity.com/paper/2606.23771/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 22 references that are in the map.
- First posted on arXiv 22 Jun 2026.

## Builds on (most cited first, 8 of 8)

- [emg2qwerty: A Large Dataset with Baselines for Touch Typing using Surface Electromyography](https://mlsanity.com/paper/2410.20081/) (2024)
- [Integrated Sensing and Communication with mmWave Massive MIMO: A Compressed Sampling Perspective](https://mlsanity.com/paper/2201.05766/) (2022)
- [DISC: a Dataset for Integrated Sensing and Communication in mmWave Systems](https://mlsanity.com/paper/2306.09469/) (2023)
- [SPARCS: A Sparse Recovery Approach for Integrated Communication and Human Sensing in mmWave Systems](https://mlsanity.com/paper/2205.03263/) (2022)
- [Cross-layer Integrated Sensing and Communication: A Joint Industrial and Academic Perspective](https://mlsanity.com/paper/2505.10933/) (2025)
- [Toward Interactive Multi-User Extended Reality Using Millimeter-Wave Networking](https://mlsanity.com/paper/2501.08100/) (2025)
- [Performance Analysis of 5G FR2 (mmWave) Downlink 256QAM on Commercial 5G Networks](https://mlsanity.com/paper/2502.03416/) (2025)
- [Millimeter-Wave Gesture Recognition in ISAC: Does Reducing Sensing Airtime Hamper Accuracy?](https://mlsanity.com/paper/2601.10733/) (2026)

## Abstract (arXiv)

Extended Reality (XR) presents a challenging use case for 5G and 6G networks, requiring high data-rates and lowlatency communication to deliver a truly immersive experience. Moreover, in order to seamlessly translate physical actions to the virtual world, accurate gesture recognition and pose estimation are required. Current XR interaction solutions based on handheld controllers and cameras cannot easily capture full-body poses, inhibit the free use of hands, and require good visibility and a clear line of sight. In this work, we propose a multimodal sensing architecture for XR that combines 5G MillimeterWave (mmWave) Integrated sensing and communication (ISAC) and surface electromyography (sEMG) signals. 5G mmWave ISAC cannot only be used to deliver content wirelessly to the Head-mounted display (HMD), but also the same communication signals can be used to derive coarse body-level gestures and poses of the user, to support real-time avatar control. For fine-grained finger-level gestures, our architecture leverages lightweight sEMG sensors that capture forearm muscle activity. To illustrate the need of both modalities, we present evaluations of both sensing technologies. At the body level (5G), our architecture relies on power-per-beam-pair (PPBP), which can be computed from standard beam management or beam sweeping procedures of the 5G NR standard. PPBP-based sensing achieves 82.2±5.9% average accuracy when evaluated on users not seen during training. For fine-grained finger-level interactions, we show that surface electromyography (sEMG) carries strong discriminative information achieving consistent promising performance across different movement settings. Thus, combining the two modalities enables multi-scale gesture recognition, at the body level via existing 5G signals and finger level via lightweight sEMG sensors, forming a complete XR framework.