# ContextCodec: Content-Focused Context Guidance for Ultra-Low Bitrate Speech Coding

arXiv: https://arxiv.org/abs/2606.10591 · HTML: https://mlsanity.com/paper/2606.10591/

## Key facts

- Cited by 1 papers in the map (data through 19 Oct 2026).
- Placed in Vocoders and speech synthesis, one of 11 directions in Speech and audio language models.
- Builds on 29 of its 38 references that are in the map.
- First posted on arXiv 9 Jun 2026.

## Placed in

- Speech and audio language models / [Vocoders and speech synthesis](https://mlsanity.com/speech/vocoders-and-speech-synthesis/)

## Builds on (most cited first, 29 of 29)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [AudioLM: a Language Modeling Approach to Audio Generation](https://mlsanity.com/paper/2209.03143/) (2022)
- [Moshi: a speech-text foundation model for real-time dialogue](https://mlsanity.com/paper/2410.00037/) (2024)
- [Joint Autoregressive and Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/1809.02736/) (2018)
- [High-Fidelity Audio Compression with Improved RVQGAN](https://mlsanity.com/paper/2306.06546/) (2023)
- [Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modules](https://mlsanity.com/paper/2001.01568/) (2020)
- [Finite Scalar Quantization: VQ-VAE Made Simple](https://mlsanity.com/paper/2309.15505/) (2023)
- [NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models](https://mlsanity.com/paper/2403.03100/) (2024)
- [Channel-wise Autoregressive Entropy Models for Learned Image Compression](https://mlsanity.com/paper/2007.08739/) (2020)
- [HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codec](https://mlsanity.com/paper/2305.02765/) (2023)
- [LPCNet: Improving Neural Speech Synthesis Through Linear Prediction](https://mlsanity.com/paper/1810.11846/) (2018)
- [AISHELL-3: A Multi-speaker Mandarin TTS Corpus and the Baselines](https://mlsanity.com/paper/2010.11567/) (2020)
- [Qwen3-TTS Technical Report](https://mlsanity.com/paper/2601.15621/) (2026)
- [Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model](https://mlsanity.com/paper/2408.17175/) (2024)
- [SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Sound](https://mlsanity.com/paper/2405.00233/) (2024)
- [SNAC: Multi-Scale Neural Audio Codec](https://mlsanity.com/paper/2410.14411/) (2024)
- [Generative Speech Coding with Predictive Variance Regularization](https://mlsanity.com/paper/2102.09660/) (2021)
- [FreeCodec: A disentangled neural speech codec with fewer tokens](https://mlsanity.com/paper/2412.01053/) (2024)
- [SoCodec: A Semantic-Ordered Multi-Stream Speech Codec for Efficient Language Model Based Text-to-Speech Synthesis](https://mlsanity.com/paper/2409.00933/) (2024)
- [SecoustiCodec: Cross-Modal Aligned Streaming Single-Codecbook Speech Codec](https://mlsanity.com/paper/2508.02849/) (2025)
- [Universal Speech Token Learning via Low-Bitrate Neural Codec and Pretrained Representations](https://mlsanity.com/paper/2503.12115/) (2025)

## Built on by (most cited first, 1 of 1)

- [ReLMCodec: Designing Predictable Speech Tokens from Pre-Quantization Phoneme Structure](https://mlsanity.com/paper/2608.08286/) (2026)

## Abstract (arXiv)

Neural speech codecs enable low-bitrate speech communication, yet at ultra-low bitrates (< 1000 bps) preserving perceptual quality and intelligibility is challenging. Existing designs often prioritize acoustic details, leaving limited capacity for the core linguistic message under tight bitrate constraints. To address this, we propose ContextCodec, a codec that transmits content-focused context features to explicitly guide reconstruction. ContextCodec adopts a dual-branch encoder that decouples acoustic details from content-focused context. The context branch is trained with a CLIP-style contrastive loss that aligns context features with phoneme indices, reducing paralinguistic leakage. During decoding, these features are injected at each decoding stage for explicit guidance. In addition, we introduce a lightweight autoregressive latent refinement module. Experiments show a strong quality-intelligibility trade-off down to 500 bps, with an RTF of 0.4886 on a typical mobile CPU.