# Scaling and Distilling Transformer Models for sEMG

arXiv: https://arxiv.org/abs/2507.22094 · HTML: https://mlsanity.com/paper/2507.22094/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 65 references that are in the map.
- First posted on arXiv 29 Jul 2025.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Efficiently Scaling Transformer Inference](https://mlsanity.com/paper/2211.05102/) (2022)
- [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019)
- [Decoupled Knowledge Distillation](https://mlsanity.com/paper/2203.08679/) (2022)
- [Cross-Layer Distillation with Semantic Calibration](https://mlsanity.com/paper/2012.03236/) (2020)
- [Understanding and Improving Knowledge Distillation](https://mlsanity.com/paper/2002.03532/) (2020)
- [DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Models](https://mlsanity.com/paper/2305.17651/) (2023)
- [Sequence-to-Sequence Speech Recognition with Time-Depth Separable Convolutions](https://mlsanity.com/paper/1904.02619/) (2019)
- [FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learning](https://mlsanity.com/paper/2207.00555/) (2022)
- [emg2qwerty: A Large Dataset with Baselines for Touch Typing using Surface Electromyography](https://mlsanity.com/paper/2410.20081/) (2024)
- [TraHGR: Transformer for Hand Gesture Recognition via ElectroMyography](https://mlsanity.com/paper/2203.16336/) (2022)
- [TEMGNet: Deep Transformer-based Decoding of Upperlimb sEMG for Hand Gestures Recognition](https://mlsanity.com/paper/2109.12379/) (2021)
- [EMGBench: Benchmarking Out-of-Distribution Generalization and Adaptation for Electromyography](https://mlsanity.com/paper/2410.23625/) (2024)
- [Exploring Effective Distillation of Self-Supervised Speech Models for Automatic Speech Recognition](https://mlsanity.com/paper/2210.15631/) (2022)
- [Electromyography Signal Classification Using Deep Learning](https://mlsanity.com/paper/2305.04006/) (2023)
- [Knowledge Distilled Ensemble Model for sEMG-based Silent Speech Interface](https://mlsanity.com/paper/2308.06533/) (2023)

## Built on by (most cited first, 3 of 3)

- [EgoEMG: A Multimodal Egocentric Dataset with Bilateral EMG and Vision for Hand Pose Estimation](https://mlsanity.com/paper/2605.05712/) (2026)
- [FatigueFormer: Static-Temporal Feature Fusion for Robust sEMG-Based Muscle Fatigue Recognition](https://mlsanity.com/paper/2603.26841/) (2026)
- [EVFormer: An Egocentric Vision-EMG Bidirectional Attention Model for Bimanual Hand Pose Estimation](https://mlsanity.com/paper/2610.06970/) (2026)

## Abstract (arXiv)

Surface electromyography (sEMG) signals offer a promising avenue for developing innovative human-computer interfaces by providing insights into muscular activity. However, the limited volume of training data and computational constraints during deployment have restricted the investigation of scaling up the model size for solving sEMG tasks. In this paper, we demonstrate that vanilla transformer models can be effectively scaled up on sEMG data and yield improved cross-user performance up to 110M parameters, surpassing the model size regime investigated in other sEMG research (usually <10M parameters). We show that >100M-parameter models can be effectively distilled into models 50x smaller with minimal loss of performance (<1.5% absolute). This results in efficient and expressive models suitable for complex real-time sEMG tasks in real-world environments.