mlsanityData to 7 Oct 2026

Autoregressive Speech Synthesis without Vector Quantization

arXiv 2407.08551 ↗First posted 11 Jul 2024cs.CL cs.SD eess.AScited by 65

Not on the map. Why?

Cited by 32 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present MELLE, a novel continuous-valued token based language modeling approach for text-to-speech synthesis (TTS). MELLE autoregressively generates continuous mel-spectrogram frames directly from text condition, bypassing the need for vector quantization, which is typically designed for audio compression and sacrifices fidelity compared to continuous representations. Specifically, (i) instead of cross-entropy loss, we apply regression loss with a proposed spectrogram flux loss function to model the probability distribution of the continuous-valued tokens; (ii) we have incorporated variational inference into MELLE to facilitate sampling mechanisms, thereby enhancing the output diversity and model robustness. Experiments demonstrate that, compared to the two-stage codec language model VALL-E and its variants, the single-stage MELLE mitigates robustness issues by avoiding the inherent flaws of sampling vector-quantized codes, achieves superior performance across multiple metrics, and, most importantly, offers a more streamlined paradigm. The demos of our work are provided at https://aka.ms/melle.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). Builds on 44 of its 58 references that are in the map. First posted on arXiv 11 Jul 2024.

Builds on

44 of 58 references.

References · 44

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  7. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  8. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  9. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  10. 2016Variational Inference: A Review for Statisticianscited by 1,466
  11. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  12. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
32 more
  1. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  2. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  3. 2022High Fidelity Neural Audio Compressioncited by 673
  4. 2024Autoregressive Image Generation without Vector Quantizationcited by 644
  5. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  6. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  7. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
  8. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  9. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  10. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  11. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 323
  12. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 293
  13. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 277
  14. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 233
  15. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 222
  16. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
  17. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 166
  18. 2023Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesiscited by 150
  19. 2024E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTScited by 149
  20. 2023StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Modelscited by 148
  21. 2023Audiobox: Unified Audio Generation with Natural Language Promptscited by 120
  22. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 119
  23. 2023SoundStorm: Efficient Parallel Audio Generationcited by 111
  24. 2023Libriheavy: a 50,000 hours ASR corpus with punctuation casing and contextcited by 84
  25. 2024BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of datacited by 74
  26. 2023Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Biascited by 51
  27. 2024ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reorderingcited by 50
  28. 2024CLaM-TTS: Improving Neural Codec Language Model for Zero-Shot Text-to-Speechcited by 37
  29. 2024VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignmentcited by 37
  30. 2023Not All Image Regions Matter: Masked Vector Quantization for Autoregressive Image Generationcited by 28
  31. 2024RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting for Text-to-Speech Synthesiscited by 26
  32. 2023Discrete Audio Representation as an Alternative to Mel-Spectrograms for Speaker and Speech Recognitioncited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 35; 2026: 17.

65 papers.

Sort:

By area:

  1. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 353
  2. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 282
  3. 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-trainingcited by 142
  4. 2025Baichuan-Omni-1.5 Technical Reportcited by 107
  5. 2025MotionStreamer: Streaming Motion Generation via Diffusion-based Autoregressive Model in Causal Latent Spacecited by 82
  6. 2025Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesiscited by 69
  7. 2025Baichuan-Audio: A Unified Framework for End-to-End Speech Interactioncited by 68
  8. 2025DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generationcited by 56
  9. 2024Multimodal Latent Language Modeling with Next-Token Diffusioncited by 47
  10. 2024JetFormer: An Autoregressive Generative Model of Raw Images and Textcited by 44
  11. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  12. 2025MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesiscited by 39
38 more
  1. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  2. 2024Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Surveycited by 22
  3. 2024Large Language Model Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructionscited by 17
  4. 2025Hyperspherical Latents Improve Continuous-Token Autoregressive Generationcited by 16
  5. 2025TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modelingcited by 15
  6. 2025Pseudo-Autoregressive Neural Codec Language Models for Efficient Zero-Shot Text-to-Speech Synthesiscited by 10
  7. 2024Audio-Agent: Leveraging LLMs For Audio Generation, Editing and Compositioncited by 8
  8. 2025FELLE: Autoregressive Speech Synthesis with Token-Wise Coarse-to-Fine Flow Matchingcited by 8
  9. 2024StyleTTS-ZS: Efficient High-Quality Zero-Shot Text-to-Speech Synthesis with Distilled Time-Varying Style Diffusioncited by 6
  10. 2025FlowDubber: Movie Dubbing with LLM-based Semantic-aware Learning and Flow Matching based Voice Enhancingcited by 6
  11. 2025Step-Audio-EditX Technical Reportcited by 6
  12. 2024DMOSpeech: Direct Metric Optimization via Distilled Diffusion Model in Zero-Shot Speech Synthesiscited by 5
  13. 2024Speech Synthesis From Continuous Features Using Per-Token Latent Diffusioncited by 5
  14. 2025Accelerating Flow-Matching-Based Text-to-Speech via Empirically Pruned Step Samplingcited by 5
  15. 2025Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Predictioncited by 5
  16. 2026ReStyle-TTS: Relative and Continuous Style Control for Zero-Shot Speech Synthesiscited by 5
  17. 2024SF-Speech: Straightened Flow for Zero-Shot Voice Clonecited by 4
  18. 2025OZSpeech: One-step Zero-shot Speech Synthesis with Learned-Prior-Conditioned Flow Matchingcited by 4
  19. 2025Accelerating Diffusion-based Text-to-Speech Model Training with Dual Modality Alignmentcited by 4
  20. 2025Cross-Lingual F5-TTS: Towards Language-Agnostic Voice Cloning and Speech Synthesiscited by 4
  21. 2025SoundReactor: Frame-level Online Video-to-Audio Generationcited by 4
  22. 2024Continuous Speech Tokenizer in Text To Speechcited by 3
  23. 2025Zero-Shot Streaming Text to Speech Synthesis with Transducer and Auto-Regressive Modelingcited by 3
  24. 2025StreamMel: Real-Time Zero-shot Text-to-Speech via Interleaved Continuous Autoregressive Modelingcited by 3
  25. 2025Next Tokens Denoising for Speech Synthesiscited by 3
  26. 2026Habibi: Laying the Open-Source Foundation of Unified-Dialectal Arabic Speech Synthesiscited by 3
  27. 2025Shallow Flow Matching for Coarse-to-Fine Text-to-Speech Synthesiscited by 2
  28. 2025UniTTS: An end-to-end TTS system without decoupling of acoustic and semantic informationcited by 2
  29. 2025Mitigating Hallucinations in LM-Based TTS Models via Distribution Alignment Using GFlowNetscited by 2
  30. 2025MELA-TTS: Joint transformer-diffusion model with representation alignment for speech synthesiscited by 2
  31. 2026DiFlowDubber: Discrete Flow Matching for Automated Video Dubbing via Cross-Modal Alignment and Synchronizationcited by 2
  32. 2025MARS6: A Small and Robust Hierarchical-Codec Text-to-Speech Modelcited by 1
  33. 2025Efficient Speech Language Modeling via Energy Distance in Continuous Latent Spacecited by 1
  34. 2025UniVoice: Unifying Autoregressive ASR and Flow-Matching based TTS with Large Language Modelscited by 1
  35. 2025Optimizing Input of Denoising Score Matching is Biased Towards Higher Score Normcited by 1
  36. 2026Scaling Properties of Continuous Diffusion Spoken Language Modelscited by 1
  37. 2025Capturing Conditional Dependence via Auto-regressive Diffusion Modelscited by 0
  38. 2025DiFlow-TTS: Compact and Low-Latency Zero-Shot Text-to-Speech with Discrete Flow Matchingcited by 0

50 most cited of 65 All 65 (JSON)