# FineCombo-TTS: Collaborative and Precise Controllable Speech Synthesis Using Text Descriptions and Reference Speech

arXiv: https://arxiv.org/abs/2606.19209 · HTML: https://mlsanity.com/paper/2606.19209/

## Key facts

- No citing papers in the map yet (data through 19 Oct 2026).
- Placed in Vocoders and speech synthesis, one of 11 directions in Speech and audio language models.
- Builds on 28 of its 29 references that are in the map.
- First posted on arXiv 17 Jun 2026.

## Placed in

- Speech and audio language models / [Vocoders and speech synthesis](https://mlsanity.com/speech/vocoders-and-speech-synthesis/)

## Builds on (most cited first, 28 of 28)

- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [High-Fidelity Audio Compression with Improved RVQGAN](https://mlsanity.com/paper/2306.06546/) (2023)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [MLS: A Large-Scale Multilingual Dataset for Speech Research](https://mlsanity.com/paper/2012.03411/) (2020)
- [Simple and Controllable Music Generation](https://mlsanity.com/paper/2306.05284/) (2023)
- [NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models](https://mlsanity.com/paper/2403.03100/) (2024)
- [Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision](https://mlsanity.com/paper/2302.03540/) (2023)
- [PromptTTS: Controllable Text-to-Speech with Text Descriptions](https://mlsanity.com/paper/2211.12171/) (2022)
- [Emotional Voice Conversion: Theory, Databases and ESD](https://mlsanity.com/paper/2105.14762/) (2021)
- [InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Prompt](https://mlsanity.com/paper/2301.13662/) (2023)
- [LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpus](https://mlsanity.com/paper/2305.18802/) (2023)
- [Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generation](https://mlsanity.com/paper/2106.03153/) (2021)
- [Natural language guidance of high-fidelity text-to-speech with synthetic annotations](https://mlsanity.com/paper/2402.01912/) (2024)
- [TextrolSpeech: A Text Style Control Speech Corpus With Codec Language Text-to-Speech Models](https://mlsanity.com/paper/2308.14430/) (2023)
- [EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Prompting](https://mlsanity.com/paper/2504.12867/) (2025)
- [PromptStyle: Controllable Style Transfer for Text-to-Speech with Natural Language Descriptions](https://mlsanity.com/paper/2305.19522/) (2023)
- [VoxInstruct: Expressive Human Instruction-to-Speech Generation with Unified Multilingual Codec Language Modelling](https://mlsanity.com/paper/2408.15676/) (2024)
- [Motion Flow Matching for Human Motion Synthesis and Editing](https://mlsanity.com/paper/2312.08895/) (2023)
- [Latent Space Editing in Transformer-Based Flow Matching](https://mlsanity.com/paper/2312.10825/) (2023)
- [ControlSpeech: Towards Simultaneous and Independent Zero-shot Speaker Cloning and Zero-shot Language Style Control](https://mlsanity.com/paper/2406.01205/) (2024)
- [LibriTTS-P: A Corpus with Speaking Style and Speaker Identity Prompts for Text-to-Speech and Style Captioning](https://mlsanity.com/paper/2406.07969/) (2024)
- [FleSpeech: Flexibly Controllable Speech Generation with Various Prompts](https://mlsanity.com/paper/2501.04644/) (2025)
- [Improving Language Model-Based Zero-Shot Text-to-Speech Synthesis with Multi-Scale Acoustic Prompts](https://mlsanity.com/paper/2309.11977/) (2023)

## Abstract (arXiv)

Controllable text-to-speech (TTS) has become a key research focus. However, methods based on either reference speech or text descriptions lack flexibility and precise control, and recent joint approaches remain loosely coupled, with speech modeling timbre and text controlling global style. We propose FineCombo-TTS, a unified framework for speech synthesis grounded in reference speech and guided by text descriptions, enabling flexible and precise control over acoustic attributes. Instead of explicit attribute disentanglement, we learn a unified acoustic representation and introduce a Conditional Flow Matching (CFM)-based Speech Variance Predictor to model fine-grained reference-to-target transformations guided by text descriptions. To support relative attribute control, we construct FineEdit, a structured paired dataset that explicitly encodes source-to-target attribute variations. Experiments demonstrate that our approach achieves flexible, precise, and expressive controllable TTS.