# SoniSpeech: A Large-Scale Open-Vocabulary Tri-Modal Dataset for Wearable Silent Speech Interfaces

arXiv: https://arxiv.org/abs/2608.00803 · HTML: https://mlsanity.com/paper/2608.00803/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 36 references that are in the map.
- First posted on arXiv 1 Aug 2026.

## Builds on (most cited first, 18 of 18)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization](https://mlsanity.com/paper/2212.10465/) (2022)
- [Digital Voicing of Silent Speech](https://mlsanity.com/paper/2010.02960/) (2020)
- [An Improved Model for Voicing Silent Speech](https://mlsanity.com/paper/2106.01933/) (2021)
- [Scaling Law in Neural Data: Non-Invasive Speech Decoding with 175 Hours of EEG Data](https://mlsanity.com/paper/2407.07595/) (2024)
- [TaL: a synchronised multi-speaker corpus of ultrasound tongue imaging, audio, and lip videos](https://mlsanity.com/paper/2011.09804/) (2020)
- [A Cross-Modal Approach to Silent Speech with LLM-Enhanced Recognition](https://mlsanity.com/paper/2403.05583/) (2024)
- [EchoWrist: Continuous Hand Pose Tracking and Hand-Object Interaction Recognition Using Low-Power Active Acoustic Sensing On a Wristband](https://mlsanity.com/paper/2401.17409/) (2024)
- [Ring-a-Pose: A Ring for Continuous Hand Pose Tracking](https://mlsanity.com/paper/2404.12980/) (2024)
- [ActSonic: Recognizing Everyday Activities from Inaudible Acoustic Wave Around the Body](https://mlsanity.com/paper/2404.13924/) (2024)
- [WatchHand: Enabling Continuous Hand Pose Tracking On Off-the-Shelf Smartwatches](https://mlsanity.com/paper/2602.21610/) (2026)
- [EyeEcho: Continuous and Low-power Facial Expression Tracking on Glasses](https://mlsanity.com/paper/2402.12388/) (2024)
- [SpellRing: Recognizing Continuous Fingerspelling in American Sign Language using a Ring](https://mlsanity.com/paper/2502.10830/) (2025)
- [MunchSonic: Tracking Fine-grained Dietary Actions through Active Acoustic Sensing on Eyeglasses](https://mlsanity.com/paper/2405.21004/) (2024)
- [SonicID: User Identification on Smart Glasses with Acoustic Sensing](https://mlsanity.com/paper/2406.08273/) (2024)
- [EchoForce: Continuous Grip Force Estimation from Skin Deformation Using Active Acoustic Sensing on a Wristband](https://mlsanity.com/paper/2507.20437/) (2025)

## Abstract (arXiv)

Wearable silent speech interfaces (SSIs) are limited to small, closed vocabularies. Approaches achieving larger vocabularies require obtrusive hardware such as facial electrodes. We present SoniSpeech, the first large-scale, open-vocabulary, trimodal dataset for wearable SSI using acoustic-sensing eyewear. It contains 34 hours across 18,000 utterances with three synchronized modalities: ultrasound echo profiles, voiced audio, and frontal video, in both voiced and silent modes. The corpus draws from the SODA dialogue dataset, providing contemporary conversational English with 5,356 unique words and full phoneme coverage. A CTC-based ResNet-34 baseline achieves 26.3% word error rate (WER) on open-vocabulary silent speech recognition, the first benchmark for this task. Dataset is available at https://doi.org/10.7298/xjjr-9m85