Joint Residual Reweighting for Classifier Free Guidance in Flow-Matching Zero-Shot TTS
On the map: Diffusion models → Audio, speech and music
Abstract (arXiv)
Classifier-free guidance (CFG) is widely used in flow-matching-based zero-shot text-to-speech (TTS), where generation is conditioned on text content and a speech prompt. Standard CFG uses a single guidance weight for their joint conditional effect, while branch-selective guidance emphasizes text or speaker conditioning and can introduce a trade-off between text accuracy and speaker similarity. In this paper, we revisit CFG under independently masked conditions and decompose the guidance field into text, speaker, and joint residuals. We show that condition-specific branch differences couple the joint residual with the corresponding text or speaker residual under a shared weight. Trajectory analysis further shows that the joint residual varies over flow time and contains information that cannot be represented by reweighting the text and speaker residuals alone. Based on these observations, we propose joint residual reweighting, which assigns independent weights to the three residuals. Experiments on F5-TTS, CosyVoice2, and GLM-TTS across three evaluation sets show overall improvements in speaker similarity and text accuracy over the default CFG settings without retraining.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 17 of its 21 references that are in the map. First posted on arXiv 24 Jun 2026.
Builds on
17 of 21 references.
Closest ancestors · 6
Same direction, newest first
- 2025Unsupervised Single-Channel Audio Separation with Diffusion Source Priors8 Dec
- 2025Cross-Lingual F5-TTS: Towards Language-Agnostic Voice Cloning and Speech Synthesis18 Sep
- 2025MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesis26 Feb
- 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models13 Dec
- 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching9 Oct
- 2023Matcha-TTS: A fast TTS architecture with conditional flow matching6 Sep
Other references · 11
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
- 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 300
- 2024E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTScited by 152
- 2022Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognitioncited by 144
- 2021Large-scale Self-Supervised Speech Representation Learning for Automatic Speaker Verificationcited by 78
- 2023Adaptive Guidance: Training-free Acceleration of Conditional Diffusion Modelscited by 31
- 2023VoiceLDM: Text-to-Speech with Environmental Contextcited by 23
- 2024DualSpeech: Enhancing Speaker-Fidelity and Text-Intelligibility Through Dual Classifier-Free Guidancecited by 11
- 2025GLM-TTS Technical Reportcited by 9
- 2025Selective Classifier-free Guidance for Zero-shot Text-to-speechcited by 4
- 2025DMP-TTS: Disentangled multi-modal Prompting for Controllable Text-to-Speech with Chained Guidancecited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Towards Flow-Matching-based TTS without Classifier-Free Guidance
- 2026Enhancing Flow Matching with A Unified Guidance Framework for Efficient and Robust Speech Synthesis
- 2025DiFlow-TTS: Compact and Low-Latency Zero-Shot Text-to-Speech with Discrete Flow Matching
- 2025Flamed-TTS: Flow Matching Attention-Free Models for Efficient Generating and Dynamic Pacing Zero-shot Text-to-Speech
- 2026RobustSpeechFlow: Learning Robust Text-to-Speech Trajectories via Augmentation-based Contrastive Flow Matching
- 2025OZSpeech: One-step Zero-shot Speech Synthesis with Learned-Prior-Conditioned Flow Matching
- 2026Align, Integrate, and Fire: Efficient Token-Level Alignment for Zero-Shot SpeechLLMs
- 2024An Investigation of Noise Robustness for Flow-Matching-Based Zero-Shot TTS