mlsanityData to 7 Oct 2026

Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidance

arXiv 2502.05236 ↗First posted 7 Feb 2025cs.SD cs.AI cs.LGcited by 26

On the map: LLM alignment / RLHF / preference optimisation → Preference alignment for recommenders and RAG

Abstract (arXiv)

While autoregressive speech token generation models produce speech with remarkable variety and naturalness, their inherent lack of controllability often results in issues such as hallucinations and undesired vocalizations that do not conform to conditioning inputs. We introduce Koel-TTS, a suite of enhanced encoder-decoder Transformer TTS models that address these challenges by incorporating preference alignment techniques guided by automatic speech recognition and speaker verification models. Additionally, we incorporate classifier-free guidance to further improve synthesis adherence to the transcript and reference speaker audio. Our experiments demonstrate that these optimizations significantly enhance target speaker similarity, intelligibility, and naturalness of synthesized speech. Notably, Koel-TTS directly maps text and context audio to acoustic tokens, and on the aforementioned metrics, outperforms state-of-the-art TTS models, despite being trained on a significantly smaller dataset. Audio samples and demos are available on our website.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in Preference alignment for recommenders and RAG, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 47 of its 51 references that are in the map. First posted on arXiv 7 Feb 2025.

Builds on

47 of 51 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Preference Alignment Improves Language Model-Based TTS19 Sep
  2. 2024SpeechAlign: Aligning Speech Generation to Human Preferences8 Apr

Other references · 45

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2022Classifier-Free Diffusion Guidancecited by 5,717
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  9. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  10. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  11. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  12. 2022High Fidelity Neural Audio Compressioncited by 673
33 more
  1. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  2. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  3. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  4. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  5. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 425
  6. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  7. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  8. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 353
  9. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 293
  10. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 282
  11. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 277
  12. 2024XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Modelcited by 174
  13. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 166
  14. 2024E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTScited by 149
  15. 2023StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Modelscited by 148
  16. 2024Nemotron-4 340B Technical Reportcited by 143
  17. 2024A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Modelscited by 98
  18. 2023Stay on topic with Classifier-Free Guidancecited by 90
  19. 2023SpeechX: Neural Codec Language Model as a Versatile Speech Transformercited by 64
  20. 2021TitaNet: Neural Model for speaker representation with 1D Depth-wise separable convolutions and global contextcited by 61
  21. 2021Hi-Fi Multi-Speaker English TTS Datasetcited by 60
  22. 2023TorchAudio-Squim: Reference-less Speech Quality and Intelligibility measures in TorchAudiocited by 56
  23. 2023Efficient Sequence Transduction by Jointly Predicting Tokens and Durationscited by 56
  24. 2024ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reorderingcited by 50
  25. 2021One TTS Alignment To Rule Them Allcited by 47
  26. 2024Improving Robustness of LLM-based Speech Synthesis by Learning Monotonic Alignmentcited by 21
  27. 2023ACE-VC: Adaptive and Controllable Voice Conversion using Explicitly Disentangled Self-supervised Speech Representationscited by 15
  28. 2023CML-TTS A Multilingual Dataset for Speech Synthesis in Low-Resource Languagescited by 15
  29. 2024Low Frame-rate Speech Codec: a Codec Designed for Fast High-quality Speech LLM Training and Inferencecited by 11
  30. 2024Can Large Language Model Summarizers Adapt to Diverse Scientific Communication Goals?cited by 6
  31. 2023SelfVC: Voice Conversion With Iterative Refinement using Self Transformationscited by 5
  32. 2024Bahasa Harmony: A Comprehensive Dataset for Bahasa Text-to-Speech Synthesis with Discrete Codec Modeling of EnGen-TTScited by 1
  33. 2024Classifier-free guidance in LLMs Safetycited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 13; 2026: 13.

26 papers.

Sort:

By area:

  1. 2025OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLMcited by 53
  2. 2025SALM-Duplex: Efficient and Direct Duplex Modeling for Speech-to-Speech Language Modelcited by 32
  3. 2025F5R-TTS: Improving Flow-Matching based Text-to-Speech with Group Relative Policy Optimizationcited by 24
  4. 2025NanoCodec: Towards High-Quality Ultra Fast Speech LLM Inferencecited by 14
  5. 2025UALM: Unified Audio Language Model for Understanding, Generation and Reasoningcited by 10
  6. 2025DMOSpeech 2: Reinforcement Learning for Duration Prediction in Metric-Optimized Speech Synthesiscited by 7
  7. 2025Multi-Metric Preference Alignment for Generative Speech Restorationcited by 7
  8. 2026Bagpiper: Solving Open-Ended Audio Tasks via Rich Captionscited by 4
  9. 2025Analyzing and Improving Speaker Similarity Assessment for Speech Synthesiscited by 3
  10. 2026VoXtream2: Full-stream TTS with dynamic speaking rate controlcited by 3
  11. 2026FlowTTS-GRPO: Online Reinforcement Learning with Multi-Objective Reward Optimization for Flow-Matching Based Text-to-Speechcited by 3
  12. 2026VoiceChat-TTS: A Low-Latency Continuous Speech Synthesis Model for Interactive Agentscited by 3
14 more
  1. 2026EmoTra-TTS: Smooth Intra-Utterance Emotion Transitions for Speech Synthesiscited by 2
  2. 2025Align2Speak: Improving TTS for Low Resource Languages via ASR-Guided Online Preference Optimizationcited by 1
  3. 2025Vox-Evaluator: Enhancing Stability and Fidelity for Zero-shot TTS with A Multi-Level Evaluatorcited by 1
  4. 2026MagpieTTS-LF: Inference-Time Long-Form Speech Generation Without Training on Long-Form datacited by 1
  5. 2026Post-Training Speech Enhancement Language Models with Perceptual Rewardscited by 1
  6. 2025Frame-Stacked Local Transformers For Efficient Multi-Codebook Speech Generationcited by 0
  7. 2025Cross-modal Consistency Guidance for Robust Emotion Control in Auto-Regressive TTS Modelscited by 0
  8. 2025The Impact of Prosodic Segmentation on Speech Synthesis of Spontaneous Speechcited by 0
  9. 2026Reliable Neural-Codec Text-to-Speech by ASR Self-Verification and Distillation: Near-Zero Catastrophic Failures Across Models and Codecscited by 0
  10. 2026Diagnose, Then Refine: A Closed-Loop TTS System with AudioLLM-Guided Correctioncited by 0
  11. 2026Ready to Speak: Aligning LLMs for TTS-Friendly Text Generationcited by 0
  12. 2026Learnable Classifier-Free Guidance Null Embeddings for Enhanced Controllable Speech Synthesiscited by 0
  13. 2026All In Good Time: Causality-Aware Framework for LLM-Based Simultaneous Speech-to-Speech Translationcited by 0
  14. 2026EmAvatar: Multimodal Empathetic Response Generation via Conflict Resolution and Expressive Guidancecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LatinX: Aligning a Multilingual TTS Model with Direct Preference Optimization
  2. 2025Data-efficient Targeted Token-level Preference Optimization for LLM-based Text-to-Speech
  3. 2025MELA-TTS: Joint transformer-diffusion model with representation alignment for speech synthesis
  4. 2026Stride-k Subsampling: Train-Free Audio Token Reduction for Whisper
  5. 2025Language Model Based Text-to-Audio Generation: Anti-Causally Aligned Collaborative Residual Transformers
  6. 2026Qwen-Audio-3.0-TTS: Freely Controllable and Highly Robust Speech Synthesis with Multi-Stage Training Paradigm
  7. 2025Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens
  8. 2026The Evolving Bottleneck in Speech Generation: Interface Co-design and Staged Alignment from CosyVoice to Qwen-Audio-3.0-TTS ↗