mlsanityData to 14 Oct 2026

Continuous Speech Tokenizer in Text To Speech

arXiv 2410.17081 ↗First posted 22 Oct 2024cs.SD cs.CL eess.AScited by 3

Not on the map. Why?

Abstract (arXiv)

The fusion of speech and language in the era of large language models has garnered significant attention. Discrete speech token is often utilized in text-to-speech tasks for speech compression and portability, which is convenient for joint training with text and have good compression efficiency. However, we found that the discrete speech tokenizer still suffers from information loss. Therefore, we propose a simple yet effective continuous speech tokenizer named Cont-SPT, and a text-to-speech model based on continuous speech tokens. Our results show that the speech language model based on the continuous speech tokenizer has better continuity and higher estimated Mean Opinion Scores (MoS). This enhancement is attributed to better information preservation rate of the continuous speech tokenizer across both low and high frequencies in the frequency domain. The code and resources for Cont-SPT can be found in https://github.com/Yixing-Li/Continuous-Speech-Tokenizer

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 21 of its 37 references that are in the map. First posted on arXiv 22 Oct 2024.

Builds on

21 of 37 references.

References · 21

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  5. 2020Efficient Transformers: A Surveycited by 790
  6. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  7. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  8. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 411
  9. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  10. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
  11. 2021NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasetscited by 192
  12. 2019MOSNet: Deep Learning based Objective Assessment for Voice Conversioncited by 127
9 more
  1. 2023UniAudio: An Audio Foundation Model Toward Universal Audio Generationcited by 127
  2. 2023HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codeccited by 97
  3. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  4. 2024Autoregressive Speech Synthesis without Vector Quantizationcited by 65
  5. 2023SpeechX: Neural Codec Language Model as a Versatile Speech Transformercited by 64
  6. 2024Automatic Speech Recognition using Advanced Deep Learning Approaches: A surveycited by 50
  7. 2023AudioDec: An Open-source Streaming High-fidelity Neural Audio Codeccited by 47
  8. 2024Towards audio language modeling -- an overviewcited by 40
  9. 2023A Survey on Efficient Training of Transformerscited by 39

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Speech Discrete Tokens or Continuous Features? A Comparative Analysis for Spoken Language Understanding in SpeechLLMscited by 4
  2. 2026HoliTok: A Continuous Holistic Tokenization with Robust Dual Capabilities of Speech Generation and Understandingcited by 2
  3. 2026Unlocking Speech-Text Compositional Powers: Instruction-Following Speech Language Models without Instruction Tuningcited by 0