# Fewer-token Neural Speech Codec with Time-invariant Codes

arXiv: https://arxiv.org/abs/2310.00014 · HTML: https://mlsanity.com/paper/2310.00014/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Sep 2023.

## Built on by (most cited first, 22 of 22)

- [VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction](https://mlsanity.com/paper/2501.01957/) (2025)
- [Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM](https://mlsanity.com/paper/2411.00774/) (2024)
- [WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling](https://mlsanity.com/paper/2408.16532/) (2024)
- [Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens](https://mlsanity.com/paper/2503.01710/) (2025)
- [Recent Advances in Speech Language Models: A Survey](https://mlsanity.com/paper/2410.03751/) (2024)
- [Recent Advances in Discrete Speech Tokens: A Review](https://mlsanity.com/paper/2502.06490/) (2025)
- [From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalities](https://mlsanity.com/paper/2412.11694/) (2024)
- [FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networks](https://mlsanity.com/paper/2502.04465/) (2025)
- [ControlSpeech: Towards Simultaneous and Independent Zero-shot Speaker Cloning and Zero-shot Language Style Control](https://mlsanity.com/paper/2406.01205/) (2024)
- [Fast and High-Quality Auto-Regressive Speech Synthesis via Speculative Decoding](https://mlsanity.com/paper/2410.21951/) (2024)
- [Low Frame-rate Speech Codec: a Codec Designed for Fast High-quality Speech LLM Training and Inference](https://mlsanity.com/paper/2409.12117/) (2024)
- [LSCodec: Low-Bitrate and Speaker-Decoupled Discrete Speech Codec](https://mlsanity.com/paper/2410.15764/) (2024)
- [Voice Cloning: Comprehensive Survey](https://mlsanity.com/paper/2505.00579/) (2025)
- [PSCodec: A Series of High-Fidelity Low-bitrate Neural Speech Codecs Leveraging Prompt Encoders](https://mlsanity.com/paper/2404.02702/) (2024)
- [VocBulwark: Towards Practical Generative Speech Watermarking via Additional-Parameter Injection](https://mlsanity.com/paper/2601.22556/) (2026)
- [Privacy-Preserving Robustness Verification for Neural Networks](https://mlsanity.com/paper/2607.05251/) (2026)
- [CRAW: Codec Robust Audio Watermarking](https://mlsanity.com/paper/2609.03107/) (2026)
- [CoDiff-VC: A Codec-Assisted Diffusion Model for Zero-shot Voice Conversion](https://mlsanity.com/paper/2411.18918/) (2024)
- [Speech Tokenizer is Key to Consistent Representation](https://mlsanity.com/paper/2507.06802/) (2025)
- [Two-Dimensional Quantization for Geometry-Aware Audio Coding](https://mlsanity.com/paper/2512.01537/) (2025)
- [CleanCodec: Efficient and Robust Speech Tokenization via Perceptually Guided Encoding](https://mlsanity.com/paper/2606.04418/) (2026)
- [TokenMapper: A Step Toward Interoperable Speech Token Translation](https://mlsanity.com/paper/2609.12563/) (2026)

## Abstract (arXiv)

Language model based text-to-speech (TTS) models, like VALL-E, have gained attention for their outstanding in-context learning capability in zero-shot scenarios. Neural speech codec is a critical component of these models, which can convert speech into discrete token representations. However, excessive token sequences from the codec may negatively affect prediction accuracy and restrict the progression of Language model based TTS models. To address this issue, this paper proposes a novel neural speech codec with time-invariant codes named TiCodec. By encoding and quantizing time-invariant information into a separate code, TiCodec can reduce the amount of frame-level information that needs encoding, effectively decreasing the number of tokens as codes of speech. Furthermore, this paper introduces a time-invariant encoding consistency loss to enhance the consistency of time-invariant code within an utterance and force it to capture more global information, which can benefit the zero-shot TTS task. Experimental results demonstrate that TiCodec can not only enhance the quality of reconstruction speech with fewer tokens but also increase the similarity and naturalness, as well as reduce the word error rate of the synthesized speech by the TTS model.