# SPG-Codec: Exploring the Role and Boundaries of Semantic Priors in Ultra-Low-Bitrate Neural Speech Coding

arXiv: https://arxiv.org/abs/2604.26296 · HTML: https://mlsanity.com/paper/2604.26296/

## Key facts

- Cited by 2 papers in the map (data through 19 Oct 2026).
- Placed in Vocoders and speech synthesis, one of 11 directions in Speech and audio language models.
- Builds on 28 of its 31 references that are in the map.
- First posted on arXiv 29 Apr 2026.

## Placed in

- Speech and audio language models / [Vocoders and speech synthesis](https://mlsanity.com/speech/vocoders-and-speech-synthesis/)

## Builds on (most cited first, 28 of 28)

- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [AudioLM: a Language Modeling Approach to Audio Generation](https://mlsanity.com/paper/2209.03143/) (2022)
- [Moshi: a speech-text foundation model for real-time dialogue](https://mlsanity.com/paper/2410.00037/) (2024)
- [High-Fidelity Audio Compression with Improved RVQGAN](https://mlsanity.com/paper/2306.06546/) (2023)
- [F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching](https://mlsanity.com/paper/2410.06885/) (2024)
- [CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training](https://mlsanity.com/paper/2505.17589/) (2025)
- [E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTS](https://mlsanity.com/paper/2406.18009/) (2024)
- [WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling](https://mlsanity.com/paper/2408.16532/) (2024)
- [VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2406.05370/) (2024)
- [HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codec](https://mlsanity.com/paper/2305.02765/) (2023)
- [SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models](https://mlsanity.com/paper/2308.16692/) (2023)
- [Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model](https://mlsanity.com/paper/2408.17175/) (2024)
- [A Survey on Speech Large Language Models for Understanding](https://mlsanity.com/paper/2410.18908/) (2024)
- [SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Sound](https://mlsanity.com/paper/2405.00233/) (2024)
- [BigCodec: Pushing the Limits of Low-Bitrate Neural Speech Codec](https://mlsanity.com/paper/2409.05377/) (2024)
- [SNAC: Multi-Scale Neural Audio Codec](https://mlsanity.com/paper/2410.14411/) (2024)
- [Fewer-token Neural Speech Codec with Time-invariant Codes](https://mlsanity.com/paper/2310.00014/) (2023)
- [RepCodec: A Speech Representation Codec for Speech Tokenization](https://mlsanity.com/paper/2309.00169/) (2023)
- [DualCodec: A Low-Frame-Rate, Semantically-Enhanced Neural Audio Codec for Speech Generation](https://mlsanity.com/paper/2505.13000/) (2025)
- [SACodec: Asymmetric Quantization with Semantic Anchoring for Low-Bitrate High-Fidelity Neural Speech Codecs](https://mlsanity.com/paper/2512.20944/) (2025)

## Built on by (most cited first, 2 of 2)

- [BAMU: Bitstream-Aware Marginal-Utility Allocation for Frozen Pretrained Neural Speech Codecs](https://mlsanity.com/paper/2608.08432/) (2026)
- [UniStream: Multi-Expert Residual Vector Quantization for 48 kHz Causal Streaming Audio Coding](https://mlsanity.com/paper/2609.09866/) (2026)

## Abstract (arXiv)

Conventional neural speech codecs suffer from severe intelligibility degradation at ultra-low bitrates, where the bottleneck transitions from acoustic distortion to semantic loss. To address this issue, this paper conducts a systematic investigation into the role and fundamental limits of integrating frozen semantic priors -- specifically HuBERT and Whisper -- into neural speech coding. We introduce and quantitatively validate a novel Semantic Retirement phenomenon: while semantic constraints reduce the Word Error Rate (WER) by up to ~10% relatively at 1.5 kbps, their benefits rapidly diminish beyond 6 kbps, indicating a practical capacity boundary. We further uncover a clear trade-off between different prior types: acoustic-rich priors (HuBERT) better preserve prosodic and timbral details, whereas high-level linguistic priors (Whisper) effectively suppress phonetic hallucinations in noisy environments (reducing hallucination rates by 26 percent) and substantially narrow the generalization gap for unseen speakers. Building on these findings, we propose a bitrate-aware regulation strategy that dynamically adjusts prior strength to optimize the trade-off between semantic consistency and perceptual naturalness. Extensive experimental evaluations confirm that our approach achieves competitive intelligibility and noise robustness compared to existing baselines, offering a principled pathway toward ultra-low-bitrate generative speech coding.