# Joint Residual Reweighting for Classifier Free Guidance in Flow-Matching Zero-Shot TTS

arXiv: https://arxiv.org/abs/2606.25672 · HTML: https://mlsanity.com/paper/2606.25672/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 17 of its 21 references that are in the map.
- First posted on arXiv 24 Jun 2026.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 17 of 17)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching](https://mlsanity.com/paper/2410.06885/) (2024)
- [Seed-TTS: A Family of High-Quality Versatile Speech Generation Models](https://mlsanity.com/paper/2406.02430/) (2024)
- [CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models](https://mlsanity.com/paper/2412.10117/) (2024)
- [E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTS](https://mlsanity.com/paper/2406.18009/) (2024)
- [Matcha-TTS: A fast TTS architecture with conditional flow matching](https://mlsanity.com/paper/2309.03199/) (2023)
- [Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition](https://mlsanity.com/paper/2206.08317/) (2022)
- [Large-scale Self-Supervised Speech Representation Learning for Automatic Speaker Verification](https://mlsanity.com/paper/2110.05777/) (2021)
- [MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesis](https://mlsanity.com/paper/2502.18924/) (2025)
- [Adaptive Guidance: Training-free Acceleration of Conditional Diffusion Models](https://mlsanity.com/paper/2312.12487/) (2023)
- [VoiceLDM: Text-to-Speech with Environmental Context](https://mlsanity.com/paper/2309.13664/) (2023)
- [DualSpeech: Enhancing Speaker-Fidelity and Text-Intelligibility Through Dual Classifier-Free Guidance](https://mlsanity.com/paper/2408.14423/) (2024)
- [GLM-TTS Technical Report](https://mlsanity.com/paper/2512.14291/) (2025)
- [Cross-Lingual F5-TTS: Towards Language-Agnostic Voice Cloning and Speech Synthesis](https://mlsanity.com/paper/2509.14579/) (2025)
- [Selective Classifier-free Guidance for Zero-shot Text-to-speech](https://mlsanity.com/paper/2509.19668/) (2025)
- [Unsupervised Single-Channel Audio Separation with Diffusion Source Priors](https://mlsanity.com/paper/2512.07226/) (2025)
- [DMP-TTS: Disentangled multi-modal Prompting for Controllable Text-to-Speech with Chained Guidance](https://mlsanity.com/paper/2512.09504/) (2025)

## Built on by (most cited first, 1 of 1)

- [Accent Analogy Guidance: More Speaker Similarity at Equal Accent in Cross-Lingual Voice Cloning](https://mlsanity.com/paper/2609.29123/) (2026)

## Abstract (arXiv)

Classifier-free guidance (CFG) is widely used in flow-matching-based zero-shot text-to-speech (TTS), where generation is conditioned on text content and a speech prompt. Standard CFG uses a single guidance weight for their joint conditional effect, while branch-selective guidance emphasizes text or speaker conditioning and can introduce a trade-off between text accuracy and speaker similarity. In this paper, we revisit CFG under independently masked conditions and decompose the guidance field into text, speaker, and joint residuals. We show that condition-specific branch differences couple the joint residual with the corresponding text or speaker residual under a shared weight. Trajectory analysis further shows that the joint residual varies over flow time and contains information that cannot be represented by reweighting the text and speaker residuals alone. Based on these observations, we propose joint residual reweighting, which assigns independent weights to the three residuals. Experiments on F5-TTS, CosyVoice2, and GLM-TTS across three evaluation sets show overall improvements in speaker similarity and text accuracy over the default CFG settings without retraining.