# AutoStyle-TTS: Retrieval-Augmented Generation based Automatic Style Matching Text-to-Speech Synthesis

arXiv: https://arxiv.org/abs/2504.10309 · HTML: https://mlsanity.com/paper/2504.10309/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Conversational and speech RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 22 of its 28 references that are in the map.
- First posted on arXiv 14 Apr 2025.

## Placed in

- Retrieval-augmented generation / [Conversational and speech RAG](https://mlsanity.com/rag/conversational-and-speech-rag/)

## Builds on (most cited first, 22 of 22)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens](https://mlsanity.com/paper/2407.05407/) (2024)
- [MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer](https://mlsanity.com/paper/2409.00750/) (2024)
- [CAM++: A Fast and Efficient Network for Speaker Verification Using Context-Aware Masking](https://mlsanity.com/paper/2303.00332/) (2023)
- [EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis](https://mlsanity.com/paper/2308.05725/) (2023)
- [InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Prompt](https://mlsanity.com/paper/2301.13662/) (2023)
- [M3ED: Multi-modal Multi-scene Multi-label Emotional Dialogue Database](https://mlsanity.com/paper/2205.10237/) (2022)
- [Retrieval-Augmented Text-to-Audio Generation](https://mlsanity.com/paper/2309.08051/) (2023)
- [VoxInstruct: Expressive Human Instruction-to-Speech Generation with Unified Multilingual Codec Language Modelling](https://mlsanity.com/paper/2408.15676/) (2024)
- [SimpleSpeech 2: Towards Simple and Efficient Text-to-Speech with Flow-based Scalar Latent Transformer Diffusion Models](https://mlsanity.com/paper/2408.13893/) (2024)
- [Leveraging Crowdsourcing Data For Deep Active Learning - An Application: Learning Intents in Alexa](https://mlsanity.com/paper/1803.04223/) (2018)
- [Retrieval Augmented Generation in Prompt-based Text-to-Speech Synthesis with Context-Aware Contrastive Language-Audio Pretraining](https://mlsanity.com/paper/2406.03714/) (2024)
- [Controlling Emotion in Text-to-Speech with Natural Language Prompts](https://mlsanity.com/paper/2406.06406/) (2024)
- [RAG-based Question Answering over Heterogeneous Data and Text](https://mlsanity.com/paper/2412.07420/) (2024)
- [Improving Speech Prosody of Audiobook Text-to-Speech Synthesis with Acoustic and Textual Contexts](https://mlsanity.com/paper/2211.02336/) (2022)
- [Clustering is Efficient for Approximate Maximum Inner Product Search](https://mlsanity.com/paper/1507.05910/) (2015)
- [U-Style: Cascading U-nets with Multi-level Speaker and Style Modeling for Zero-Shot Voice Cloning](https://mlsanity.com/paper/2310.04004/) (2023)
- [StyleFusion TTS: Multimodal Style-control and Enhanced Feature Fusion for Zero-shot Text-to-speech Synthesis](https://mlsanity.com/paper/2409.15741/) (2024)
- [Expressive Prompting: Improving Emotion Intensity and Speaker Consistency in Zero-Shot TTS](https://mlsanity.com/paper/2409.18512/) (2024)

## Built on by (most cited first, 1 of 1)

- [Modeling Sarcastic Speech: Semantic and Prosodic Cues in a Speech Synthesis Framework](https://mlsanity.com/paper/2510.07096/) (2025)

## Abstract (arXiv)

With the advancement of speech synthesis technology, users have higher expectations for the naturalness and expressiveness of synthesized speech. But previous research ignores the importance of prompt selection. This study proposes a text-to-speech (TTS) framework based on Retrieval-Augmented Generation (RAG) technology, which can dynamically adjust the speech style according to the text content to achieve more natural and vivid communication effects. We have constructed a speech style knowledge database containing high-quality speech samples in various contexts and developed a style matching scheme. This scheme uses embeddings, extracted by Llama, PER-LLM-Embedder,and Moka, to match with samples in the knowledge database, selecting the most appropriate speech style for synthesis. Furthermore, our empirical research validates the effectiveness of the proposed method. Our demo can be viewed at: https://thuhcsi.github.io/icme2025-AutoStyle-TTS