Speech Recognition Model Improves Text-to-Speech Synthesis using Fine-Grained Reward
Not on the map. Why?
Abstract (arXiv)
Recent advances in text-to-speech (TTS) have enabled models to clone arbitrary unseen speakers and synthesize high-quality, natural-sounding speech. However, evaluation methods lag behind: typical mean opinion score (MOS) estimators perform regression over entire utterances, while failures usually occur in a few problematic words. We observe that encoder-decoder ASR models (e.g., Whisper) surface word-level mismatches between speech and text via cross-attention, providing a fine-grained reward signal. Building on this, we introduce Word-level TTS Alignment by ASR-driven Attentive Reward (W3AR). Without explicit reward annotations, W3AR uses attention from a pre-trained ASR model to drive finer-grained alignment and optimization of sequences predicted by a TTS model. Experiments show that W3AR improves the quality of existing TTS systems and strengthens zero-shot robustness on unseen speakers. More broadly, our results suggest a simple recipe for generative modeling: understanding models can act as evaluators, delivering informative, fine-grained feedback for optimization.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 23 of its 28 references that are in the map. First posted on arXiv 12 Nov 2025.
Builds on
23 of 28 references.
References · 23
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
- 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
- 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
- 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
- 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speechcited by 447
- 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
- 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 399
- 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
- 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 229
- 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 204
- 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 161
- 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 122
11 more
- 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 120
- 2025Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokenscited by 113
- 2023AudioDec: An Open-source Streaming High-fidelity Neural Audio Codeccited by 47
- 2024SpeechAlign: Aligning Speech Generation to Human Preferencescited by 47
- 2024Attend First, Consolidate Later: On the Importance of Attention in Different LLM Layerscited by 25
- 2024VALL-T: Decoder-Only Generative Transducer for Robust and Decoding-Controllable Text-to-Speechcited by 23
- 2024Improving Robustness of LLM-based Speech Synthesis by Learning Monotonic Alignmentcited by 22
- 2024Preference Alignment Improves Language Model-Based TTScited by 21
- 2024Enhancing Zero-shot Text-to-Speech Synthesis with Human Feedbackcited by 20
- 2024Robust Zero-Shot Text-to-Speech Synthesis with Reverse Inference Optimizationcited by 11
- 2024Self-Taught Recognizer: Toward Unsupervised Adaptation for Speech Foundation Modelscited by 8
Built on by
Citing papers per year: 2026: 1.
1 paper.