# Real-time speech enhancement in noise for throat microphone using neural audio codec as foundation model

arXiv: https://arxiv.org/abs/2508.02974 · HTML: https://mlsanity.com/paper/2508.02974/

## Key facts

- Cited by 1 papers in the map (data through 19 Oct 2026).
- Placed in Speech enhancement and noise suppression, one of 11 directions in Speech and audio language models.
- Builds on 17 of its 24 references that are in the map.
- First posted on arXiv 5 Aug 2025.

## Placed in

- Speech and audio language models / [Speech enhancement and noise suppression](https://mlsanity.com/speech/speech-enhancement-and-noise-suppression/)

## Builds on (most cited first, 17 of 17)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Moshi: a speech-text foundation model for real-time dialogue](https://mlsanity.com/paper/2410.00037/) (2024)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [NeMo: a toolkit for building AI applications using Neural Modules](https://mlsanity.com/paper/1909.09577/) (2019)
- [NORESQA: A Framework for Speech Quality Assessment using Non-Matching References](https://mlsanity.com/paper/2109.08125/) (2021)
- [Generative Speech Foundation Model Pretraining for High-Quality Speech Extraction and Restoration](https://mlsanity.com/paper/2409.16117/) (2024)
- [TRAMBA: A Hybrid Transformer and Mamba Architecture for Practical Audio and Bone Conduction Speech Super Resolution and Enhancement on Mobile and Wearable Platforms](https://mlsanity.com/paper/2405.01242/) (2024)
- [Time-Domain Multi-modal Bone/air Conducted Speech Enhancement](https://mlsanity.com/paper/1911.09847/) (2019)
- [Regeneration Learning: A Learning Paradigm for Data Generation](https://mlsanity.com/paper/2301.08846/) (2023)
- [Vibravox: A Dataset of French Speech Captured with Body-conduction Audio Sensors](https://mlsanity.com/paper/2407.11828/) (2024)
- [EBEN: Extreme bandwidth extension network applied to speech signals captured with noise-resilient body-conduction microphones](https://mlsanity.com/paper/2210.14090/) (2022)
- [Configurable EBEN: Extreme Bandwidth Extension Network to enhance body-conducted speech capture](https://mlsanity.com/paper/2303.10008/) (2023)
- [A Closer Look at Neural Codec Resynthesis: Bridging the Gap between Codec and Waveform Generation](https://mlsanity.com/paper/2410.22448/) (2024)
- [Low-Complexity Own Voice Reconstruction for Hearables with an In-Ear Microphone](https://mlsanity.com/paper/2409.04136/) (2024)
- [French Listening Tests for the Assessment of Intelligibility, Quality, and Identity of Body-Conducted Speech Enhancement](https://mlsanity.com/paper/2506.04495/) (2025)

## Built on by (most cited first, 1 of 1)

- [Modeling strategies for speech enhancement in the latent space of a neural audio codec](https://mlsanity.com/paper/2510.26299/) (2025)

## Abstract (arXiv)

We present a real-time speech enhancement demo using speech captured with a throat microphone. This demo aims to showcase the complete pipeline, from recording to deep learning-based post-processing, for speech captured in noisy environments with a body-conducted microphone. The throat microphone records skin vibrations, which naturally attenuate external noise, but this robustness comes at the cost of reduced audio bandwidth. To address this challenge, we fine-tune Kyutai's Mimi--a neural audio codec supporting real-time inference--on Vibravox, a dataset containing paired air-conducted and throat microphone recordings. We compare this enhancement strategy against state-of-the-art models and demonstrate its superior performance. The inference runs in an interactive interface that allows users to toggle enhancement, visualize spectrograms, and monitor processing latency.