# Noisy Environment Adaptation of Neural Speech Codec via Focal Mask and Noise Feature Separation

arXiv: https://arxiv.org/abs/2607.04195 · HTML: https://mlsanity.com/paper/2607.04195/

## Key facts

- No citing papers in the map yet (data through 19 Oct 2026).
- Placed in Vocoders and speech synthesis, one of 11 directions in Speech and audio language models.
- Builds on 20 of its 31 references that are in the map.
- First posted on arXiv 5 Jul 2026.

## Placed in

- Speech and audio language models / [Vocoders and speech synthesis](https://mlsanity.com/speech/vocoders-and-speech-synthesis/)

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [High-Fidelity Audio Compression with Improved RVQGAN](https://mlsanity.com/paper/2306.06546/) (2023)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [Recent Advances in Speech Language Models: A Survey](https://mlsanity.com/paper/2410.03751/) (2024)
- [An Investigation of Incorporating Mamba for Speech Enhancement](https://mlsanity.com/paper/2405.06573/) (2024)
- [SELM: Speech Enhancement Using Discrete Tokens and Language Models](https://mlsanity.com/paper/2312.09747/) (2023)
- [Discrete Audio Tokens: More Than a Survey!](https://mlsanity.com/paper/2506.10274/) (2025)
- [Neural Target Speech Extraction: An Overview](https://mlsanity.com/paper/2301.13341/) (2023)
- [GenSE: Generative Speech Enhancement via Language Models using Hierarchical Modeling](https://mlsanity.com/paper/2502.02942/) (2025)
- [FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networks](https://mlsanity.com/paper/2502.04465/) (2025)
- [Waveform Boundary Detection for Partially Spoofed Audio](https://mlsanity.com/paper/2211.00226/) (2022)
- [Speech Enhancement Using Continuous Embeddings of Neural Audio Codec](https://mlsanity.com/paper/2502.16240/) (2025)
- [GAN-Based Speech Enhancement for Low SNR Using Latent Feature Conditioning](https://mlsanity.com/paper/2410.13599/) (2024)
- [Neural Speech and Audio Coding: Modern AI Technology Meets Traditional Codecs](https://mlsanity.com/paper/2408.06954/) (2024)
- [Modeling strategies for speech enhancement in the latent space of a neural audio codec](https://mlsanity.com/paper/2510.26299/) (2025)
- [From Continuous to Discrete: Cross-Domain Collaborative General Speech Enhancement via Hierarchical Language Models](https://mlsanity.com/paper/2507.19062/) (2025)
- [Towards Bitrate-Efficient and Noise-Robust Speech Coding with Variable Bitrate RVQ](https://mlsanity.com/paper/2506.16538/) (2025)

## Abstract (arXiv)

Neural speech codec has attracted extensive attention for high-quality reconstruction at low-bitrate. However, real-world noise severely degrades its performance and hinders high-quality clean speech reconstruction. To tackle this problem, we propose FocalSE, a novel speech enhancement method that performs feature denoising, noise feature separation and noise recognition in the continuous embedding space of neural speech codecs. Specifically, we develop focal modulation-based compression and decompression to capture global context and local mutual information, and generate focal masks to recover clean feature embeddings. We then separate noise embeddings from noisy embeddings to improve denoising performance. Finally, we use ResNet1D-18 to recognize noise categories for better separation effectiveness. Extensive experiments on two standard datasets, LibriTTS and ESC50, demonstrate that our method outperforms state-of-the-art approaches under low-bitrate and low-SNR conditions.