mlsanityData to 7 Oct 2026

RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting for Text-to-Speech Synthesis

arXiv 2404.03204 ↗First posted 4 Apr 2024eess.AS cs.AI cs.CLcited by 26

On the map: LLM reasoning → Audio-language model reasoning

Abstract (arXiv)

We present RALL-E, a robust language modeling method for text-to-speech (TTS) synthesis. While previous work based on large language models (LLMs) shows impressive performance on zero-shot TTS, such methods often suffer from poor robustness, such as unstable prosody (weird pitch and rhythm/duration) and a high word error rate (WER), due to the autoregressive prediction style of language models. The core idea behind RALL-E is chain-of-thought (CoT) prompting, which decomposes the task into simpler steps to enhance the robustness of LLM-based TTS. To accomplish this idea, RALL-E first predicts prosody features (pitch and duration) of the input text and uses them as intermediate conditions to predict speech tokens in a CoT style. Second, RALL-E utilizes the predicted duration prompt to guide the computing of self-attention weights in Transformer to enforce the model to focus on the corresponding phonemes and prosody features when predicting speech tokens. Results of comprehensive objective and subjective evaluations demonstrate that, compared to a powerful baseline method VALL-E, RALL-E significantly improves the WER of zero-shot TTS from 5.6% (without reranking) and 1.7% (with reranking) to 2.5% and 1.0%, respectively. Furthermore, we demonstrate that RALL-E correctly synthesizes sentences that are hard for VALL-E and reduces the error rate from 68% to 4%.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning. Builds on 29 of its 70 references that are in the map. First posted on arXiv 4 Apr 2024.

Builds on

29 of 70 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  5. 2022Emergent Abilities of Large Language Modelscited by 2,483
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  7. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  8. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  9. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  10. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  11. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  12. 2022High Fidelity Neural Audio Compressioncited by 673
17 more
  1. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  2. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  3. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  4. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 381
  5. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  6. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  7. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 233
  8. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 222
  9. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 167
  10. 2023UniAudio: An Audio Foundation Model Toward Universal Audio Generationcited by 127
  11. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  12. 2020Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modelingcited by 65
  13. 2024ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reorderingcited by 50
  14. 2020MultiSpeech: Multi-Speaker Text to Speech with Transformercited by 42
  15. 2019Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTScited by 33
  16. 2019Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversioncited by 26
  17. 2024VALL-T: Decoder-Only Generative Transducer for Robust and Decoding-Controllable Text-to-Speechcited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 10; 2026: 3.

26 papers.

Sort:

By area:

  1. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 353
  2. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 282
  3. 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-trainingcited by 142
  4. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 119
  5. 2024Autoregressive Speech Synthesis without Vector Quantizationcited by 65
  6. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  7. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  8. 2024VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignmentcited by 37
  9. 2024Preference Alignment Improves Language Model-Based TTScited by 21
  10. 2024Enhancing Zero-shot Text-to-Speech Synthesis with Human Feedbackcited by 19
  11. 2024SimpleSpeech 2: Towards Simple and Efficient Text-to-Speech with Flow-based Scalar Latent Transformer Diffusion Modelscited by 17
  12. 2025TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modelingcited by 15
14 more
  1. 2024Robust Zero-Shot Text-to-Speech Synthesis with Reverse Inference Optimizationcited by 10
  2. 2025FELLE: Autoregressive Speech Synthesis with Token-Wise Coarse-to-Fine Flow Matchingcited by 8
  3. 2025Discrete Tokenization for Multimodal LLMs: A Comprehensive Surveycited by 7
  4. 2025Voice Cloning: Comprehensive Surveycited by 4
  5. 2025Robust and Efficient Autoregressive Speech Synthesis with Dynamic Chunk-wise Prediction Policycited by 4
  6. 2025Universal Speech Token Learning via Low-Bitrate Neural Codec and Pretrained Representationscited by 3
  7. 2025Zero-Shot Streaming Text to Speech Synthesis with Transducer and Auto-Regressive Modelingcited by 3
  8. 2024Does Current Deepfake Audio Detection Model Effectively Detect ALM-based Deepfake Audio?cited by 2
  9. 2024Text-to-Speech for Unseen Speakers via Low-Complexity Discrete Unit-Based Frame Selectioncited by 1
  10. 2024Single-stage TTS with Masked Audio Token Modeling and Semantic Knowledge Distillationcited by 1
  11. 2024Lina-Speech: Gated Linear Attention and Initial-State Tuning for Multi-Sample Prompting Text-To-Speech Synthesiscited by 1
  12. 2026Alignment-Free Text-Audiobox for Voice Dubbing and Full-Duplex Dialogue Synthesiscited by 0
  13. 2026Complex-Text Robustness Evaluation and Failure Diagnosis for Low-Resource Multilingual Text-to-Speechcited by 0
  14. 2026Repetition, Not Length: Isolating the Counting Failure in Neural Text-to-Speechcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Qwen-Audio-3.0-TTS: Freely Controllable and Highly Robust Speech Synthesis with Multi-Stage Training Paradigm
  2. 2025Listening or Reading? Evaluating Speech Awareness in Chain-of-Thought Speech-to-Text Translation
  3. 2024WavLLM: Towards Robust and Adaptive Speech Large Language Model
  4. 2025Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis
  5. 2024ProsodyFM: Unsupervised Phrasing and Intonation Control for Intelligible Speech Synthesis
  6. 2026COT-TTS: Audio Context-Aware Text-to-Speech with Chain-of-Thought Reasoning
  7. 2026CuteTTS: Efficient and High-Quality Speech Synthesis via Autoregressive Modeling of Continuous Latents
  8. 2025VocalNet-M2: Advancing Low-Latency Spoken Language Modeling via Integrated Multi-Codebook Tokenization and Multi-Token Prediction