mlsanityData to 14 Oct 2026

Eliminating stability hallucinations in llm-based tts models via attention guidance

arXiv 2509.19852 ↗First posted 24 Sep 2025cs.SD cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

This paper focuses on resolving stability hallucinations (e.g., repetitive or omitted speech) in LLM-based Text-to-Speech (TTS) models by improving and leveraging the attention mechanism. First, we analyzed the alignment mechanism between text tokens and speech tokens in LLMs. We then proposed a metric termed the Optimal Alignment Score (OAS), which employs the Viterbi algorithm to evaluate text-speech alignment quality. Subsequently, OAS was integrated into the training of CosyVoice2 to assist LLMs in learning continuous, stable alignment. Additionally, the pre-trained attention value is employed to guide the training of the student CosyVoice2 via chain-of-thought (CoT), which further reduces stability hallucinations in synthesized speech. Experiments on the Seed-TTS-Eval and CV3-Eval test sets demonstrate that the proposed methods can effectively reduce the stability hallucinations of CosyVoice2 without introducing additional negative effects. The appendix is available at https://wsmzzz.github.io/llm_attn.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 24 of its 26 references that are in the map. First posted on arXiv 24 Sep 2025.

Builds on

24 of 26 references.

References · 24

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2022High Fidelity Neural Audio Compressioncited by 687
  4. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
  5. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 361
  6. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
  7. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 288
  8. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 273
  9. 2024E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTScited by 152
  10. 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-trainingcited by 150
  11. 2022Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognitioncited by 144
  12. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 129
12 more
  1. 2024Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Modelcited by 72
  2. 2025Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesiscited by 70
  3. 2017Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attentioncited by 57
  4. 2024BigCodec: Pushing the Limits of Low-Bitrate Neural Speech Codeccited by 46
  5. 2023FunCodec: A Fundamental, Reproducible and Integrable Open-source Toolkit for Neural Speech Codeccited by 44
  6. 2025MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesiscited by 41
  7. 2024WenetSpeech4TTS: A 12,800-hour Mandarin TTS Corpus for Large Speech Generation Model Benchmarkcited by 40
  8. 2024VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignmentcited by 38
  9. 2019Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTScited by 33
  10. 2020AlignTTS: Efficient Feed-Forward Text-to-Speech System without Explicit Alignmentcited by 20
  11. 2018Forward Attention in Sequence-to-sequence Acoustic Modelling for Speech Synthesiscited by 17
  12. 2024Attention-Constrained Inference for Robust Decoder-Only Text-to-Speechcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Experience-Calibrated Contrastive Decoding for Mitigating Hallucinations in LM-Based Text-to-Speechcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Improving Robustness of LLM-based Speech Synthesis by Learning Monotonic Alignment
  2. 2025Mitigating Hallucinations in LM-Based TTS Models via Distribution Alignment Using GFlowNets
  3. 2026CTC-TTS: LLM-based dual-streaming text-to-speech with CTC alignment
  4. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning
  5. 2026LLMs as High-Dimensional Nonlinear Autoregressive Models with Attention: Training, Alignment and Inference
  6. 2025Data-efficient Targeted Token-level Preference Optimization for LLM-based Text-to-Speech
  7. 2025ALAS: An Automatic Latent Alignment Score for Audio Language Models
  8. 2026Gradient-Based Speech-to-Text Alignment for Any ASR Model: From CTC to Speech LLMs