mlsanityData to 14 Oct 2026

ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reordering

arXiv 2401.07333 ↗First posted 14 Jan 2024cs.CL cs.AI cs.SDcited by 50

Not on the map. Why?

Abstract (arXiv)

The language model (LM) approach based on acoustic and linguistic prompts, such as VALL-E, has achieved remarkable progress in the field of zero-shot audio generation. However, existing methods still have some limitations: 1) repetitions, transpositions, and omissions in the output synthesized speech due to limited alignment constraints between audio and phoneme tokens; 2) challenges of fine-grained control over the synthesized speech with autoregressive (AR) language model; 3) infinite silence generation due to the nature of AR-based decoding, especially under the greedy strategy. To alleviate these issues, we propose ELLA-V, a simple but efficient LM-based zero-shot text-to-speech (TTS) framework, which enables fine-grained control over synthesized audio at the phoneme level. The key to ELLA-V is interleaving sequences of acoustic and phoneme tokens, where phoneme tokens appear ahead of the corresponding acoustic tokens. The experimental findings reveal that our model outperforms VALL-E in terms of accuracy and delivers more stable results using both greedy and sampling-based decoding strategies. The code of ELLA-V will be open-sourced after cleanups. Audio samples are available at https://ereboas.github.io/ELLAV/.

Key facts. Cited by 50 papers in the map (data through 14 Oct 2026). Builds on 48 of its 64 references that are in the map. First posted on arXiv 14 Jan 2024.

Builds on

48 of 64 references.

References · 48

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  10. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  12. 2019The Curious Case of Neural Text Degenerationcited by 2,916
36 more
  1. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  2. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  3. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  5. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  6. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  7. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,043
  8. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  9. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  10. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  11. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
  12. 2019Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networkscited by 541
  13. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 490
  14. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  15. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speechcited by 447
  16. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 379
  17. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 324
  18. 2018WaveGlow: A Flow-based Generative Network for Speech Synthesiscited by 309
  19. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 273
  20. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 235
  21. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  22. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  23. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 169
  24. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 167
  25. 2021Diff-TTS: A Denoising Diffusion Model for Text-to-Speechcited by 157
  26. 2022ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speechcited by 149
  27. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  28. 2018Hierarchical Generative Modeling for Controllable Speech Synthesiscited by 93
  29. 2021Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidancecited by 84
  30. 2017Deep Voice: Real-time Neural Text-to-Speechcited by 83
  31. 2019Comparison of Diverse Decoding Methods from Conditional Language Modelscited by 78
  32. 2023LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPTcited by 78
  33. 2023SpeechX: Neural Codec Language Model as a Versatile Speech Transformercited by 64
  34. 2023Decoder-Only or Encoder-Decoder? Interpreting Language Model as a Regularized Encoder-Decodercited by 44
  35. 2022ItoWave: Ito Stochastic Differential Equation Is All You Need For Wave Generationcited by 5
  36. 2023Investigating the Utility of Surprisal from Large Language Models for Speech Synthesis Prosodycited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 21; 2025: 20; 2026: 9.

50 papers.

Sort:

By area:

  1. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 361
  2. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 288
  3. 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-trainingcited by 150
  4. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 122
  5. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 120
  6. 2025Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesiscited by 70
  7. 2024Autoregressive Speech Synthesis without Vector Quantizationcited by 65
  8. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 43
  9. 2025MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesiscited by 41
  10. 2024VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignmentcited by 38
  11. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  12. 2025Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidancecited by 27
38 more
  1. 2024RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting for Text-to-Speech Synthesiscited by 26
  2. 2024Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Surveycited by 23
  3. 2024Improving Robustness of LLM-based Speech Synthesis by Learning Monotonic Alignmentcited by 22
  4. 2024Fast and High-Quality Auto-Regressive Speech Synthesis via Speculative Decodingcited by 17
  5. 2024PSLM: Parallel Generation of Text and Speech with LLMs for Low-Latency Spoken Dialogue Systemscited by 14
  6. 2024Accelerating Codec-based Speech Synthesis with Multi-Token Prediction and Speculative Decodingcited by 10
  7. 2025Pseudo-Autoregressive Neural Codec Language Models for Efficient Zero-Shot Text-to-Speech Synthesiscited by 10
  8. 2025FELLE: Autoregressive Speech Synthesis with Token-Wise Coarse-to-Fine Flow Matchingcited by 8
  9. 2024E1 TTS: Simple and Fast Non-Autoregressive TTScited by 7
  10. 2024StyleTTS-ZS: Efficient High-Quality Zero-Shot Text-to-Speech Synthesis with Distilled Time-Varying Style Diffusioncited by 7
  11. 2025ZipVoice-Dialog: Non-Autoregressive Spoken Dialogue Generation with Flow Matchingcited by 7
  12. 2024KALL-E:Autoregressive Speech Synthesis with Next-Distribution Predictioncited by 5
  13. 2025Spoken Language Modeling with Duration-Penalized Self-Supervised Unitscited by 5
  14. 2024Robust and Unbounded Length Generalization in Autoregressive Transformer-Based Text-to-Speechcited by 4
  15. 2025OZSpeech: One-step Zero-shot Speech Synthesis with Learned-Prior-Conditioned Flow Matchingcited by 4
  16. 2025Robust and Efficient Autoregressive Speech Synthesis with Dynamic Chunk-wise Prediction Policycited by 4
  17. 2024Bailing-TTS: Chinese Dialectal Speech Synthesis Towards Human-like Spontaneous Representationcited by 3
  18. 2025Zero-Shot Streaming Text to Speech Synthesis with Transducer and Auto-Regressive Modelingcited by 3
  19. 2025DiSTAR: Diffusion over a Scalable Token Autoregressive Representation for Speech Generationcited by 3
  20. 2025VoiceCraft-X: Unifying Multilingual, Voice-Cloning Speech Synthesis and Speech Editingcited by 3
  21. 2026CTC-TTS: LLM-based dual-streaming text-to-speech with CTC alignmentcited by 3
  22. 2024Does Current Deepfake Audio Detection Model Effectively Detect ALM-based Deepfake Audio?cited by 2
  23. 2026Unifying Speech Recognition, Synthesis and Conversion with Autoregressive Transformerscited by 2
  24. 2026DiFlowDubber: Discrete Flow Matching for Automated Video Dubbing via Cross-Modal Alignment and Synchronizationcited by 2
  25. 2024The X-LANCE Technical Report for Interspeech 2024 Speech Processing Using Discrete Speech Unit Challengecited by 1
  26. 2024BrainCodec: Neural fMRI codec for the decoding of cognitive brain statescited by 1
  27. 2025MARS6: A Small and Robust Hierarchical-Codec Text-to-Speech Modelcited by 1
  28. 2025Efficient Speech Language Modeling via Energy Distance in Continuous Latent Spacecited by 1
  29. 2025DiTReducio: A Training-Free Acceleration for DiT-Based TTS via Progressive Calibrationcited by 1
  30. 2025Align2Speak: Improving TTS for Low Resource Languages via ASR-Guided Online Preference Optimizationcited by 1
  31. 2026WavTTS: Towards High-Quality Zero-Shot TTS via Direct Raw Waveform Modelingcited by 1
  32. 2025Accelerating Diffusion Transformer-Based Text-to-Speech with Transformer Layer Cachingcited by 0
  33. 2025Speak, Edit, Repeat: High-Fidelity Voice Editing and Zero-Shot TTS with Cross-Attentive Mambacited by 0
  34. 2026MELD: Mel-Spectrogram-Based Speech Language Modeling with Discrete Latent Variablescited by 0
  35. 2026Experience-Calibrated Contrastive Decoding for Mitigating Hallucinations in LM-Based Text-to-Speechcited by 0
  36. 2026Alignment-Free Text-Audiobox for Voice Dubbing and Full-Duplex Dialogue Synthesiscited by 0
  37. 2026X2-NativeCursor: Native-Token Text Progress Tracking for Incremental-Text Streaming Codec TTScited by 0
  38. 2026Repetition, Not Length: Isolating the Counting Failure in Neural Text-to-Speechcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens
  2. 2026TADA: A Generative Framework for Speech Modeling via Text-Acoustic Dual Alignment
  3. 2026CuteTTS: Efficient and High-Quality Speech Synthesis via Autoregressive Modeling of Continuous Latents
  4. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens
  5. 2026Align, Integrate, and Fire: Efficient Token-Level Alignment for Zero-Shot SpeechLLMs
  6. 2025HH-Codec: High Compression High-fidelity Discrete Neural Codec for Spoken Language Modeling
  7. 2025Language Model Based Text-to-Audio Generation: Anti-Causally Aligned Collaborative Residual Transformers
  8. 2025CLEAR: Continuous Latent Autoregressive Modeling for High-quality and Low-latency Speech Synthesis