# Can LLMs Understand Unvoiced Speech? Exploring EMG-to-Text Conversion with LLMs

arXiv: https://arxiv.org/abs/2506.00304 · HTML: https://mlsanity.com/paper/2506.00304/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 51 references that are in the map.
- First posted on arXiv 30 May 2025.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [Beam Search Strategies for Neural Machine Translation](https://mlsanity.com/paper/1702.01806/) (2017)
- [MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning](https://mlsanity.com/paper/2212.10773/) (2022)
- [Connecting Speech Encoder and Large Language Model for ASR](https://mlsanity.com/paper/2309.13963/) (2023)
- [Word Embeddings Are Steers for Language Models](https://mlsanity.com/paper/2305.12798/) (2023)
- [IMUPoser: Full-Body Pose Estimation using IMUs in Phones, Watches, and Earbuds](https://mlsanity.com/paper/2304.12518/) (2023)
- [Cut Your Losses in Large-Vocabulary Language Models](https://mlsanity.com/paper/2411.09009/) (2024)
- [Where Visual Speech Meets Language: VSP-LLM Framework for Efficient and Context-Aware Visual Speech Processing](https://mlsanity.com/paper/2402.15151/) (2024)
- [Digital Voicing of Silent Speech](https://mlsanity.com/paper/2010.02960/) (2020)
- [Frozen Large Language Models Can Perceive Paralinguistic Aspects of Speech](https://mlsanity.com/paper/2410.01162/) (2024)
- [SSR: Alignment-Aware Modality Connector for Speech Language Models](https://mlsanity.com/paper/2410.00168/) (2024)
- [An Improved Model for Voicing Silent Speech](https://mlsanity.com/paper/2106.01933/) (2021)
- [Silent Speech Interfaces for Speech Restoration: A Review](https://mlsanity.com/paper/2009.02110/) (2020)
- [A Cross-Modal Approach to Silent Speech with LLM-Enhanced Recognition](https://mlsanity.com/paper/2403.05583/) (2024)
- [Effect of Attention and Self-Supervised Speech Embeddings on Non-Semantic Speech Tasks](https://mlsanity.com/paper/2308.14359/) (2023)
- [Joint Beam Search Integrating CTC, Attention, and Transducer Decoders](https://mlsanity.com/paper/2406.02950/) (2024)
- [Missingness-resilient Video-enhanced Multimodal Disfluency Detection](https://mlsanity.com/paper/2406.06964/) (2024)
- [Split Adaptation for Pre-trained Vision Transformers](https://mlsanity.com/paper/2503.00441/) (2025)
- [Non-verbal Hands-free Control for Smart Glasses using Teeth Clicks](https://mlsanity.com/paper/2408.11346/) (2024)

## Built on by (most cited first, 6 of 6)

- [MAESTRO : Adaptive Sparse Attention and Robust Learning for Multimodal Dynamic Time Series](https://mlsanity.com/paper/2509.25278/) (2025)
- [Standardization of Neuromuscular Reflex Analysis -- Role of Fine-Tuned Vision-Language Model Consortium and OpenAI gpt-oss Reasoning LLM Enabled Decision Support System](https://mlsanity.com/paper/2508.12473/) (2025)
- [STEP-LLM: Generating CAD STEP Models from Natural Language with Large Language Models](https://mlsanity.com/paper/2601.12641/) (2026)
- [Modality-Aware Zero-Shot Pruning and Sparse Attention for Efficient Multimodal Edge Inference](https://mlsanity.com/paper/2604.08971/) (2026)
- [MyoSem: Aligning Electromyography with Natural-Language Action Semantics](https://mlsanity.com/paper/2606.00174/) (2026)
- [Myovox: Reading Speech from the Muscles of the Face](https://mlsanity.com/paper/2609.17548/) (2026)

## Abstract (arXiv)

Unvoiced electromyography (EMG) is an effective communication tool for individuals unable to produce vocal speech. However, most prior methods rely on paired voiced and unvoiced EMG signals, along with speech data, for EMG-to-text conversion, which is not practical for such individuals. Given the rise of large language models (LLMs) in speech recognition, we explore their potential to understand unvoiced speech. To this end, we address the challenge of learning from unvoiced EMG alone and propose a novel EMG adaptor module that maps EMG features into an LLM's input space, achieving an average word error rate (WER) of 0.49 on a closed-vocabulary unvoiced EMG-to-text task. Even with a conservative data availability of just six minutes, our approach improves performance over specialized models by nearly 20%. While LLMs have been shown to be extendable to new language modalities -- such as audio -- understanding articulatory biosignals like unvoiced EMG remains more challenging. This work takes a crucial first step toward enabling LLMs to comprehend unvoiced speech using surface EMG.