# FCTalker: Fine and Coarse Grained Context Modeling for Expressive Conversational Speech Synthesis

arXiv: https://arxiv.org/abs/2210.15360 · HTML: https://mlsanity.com/paper/2210.15360/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 35 references that are in the map.
- First posted on arXiv 27 Oct 2022.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset](https://mlsanity.com/paper/1710.03957/) (2017)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset](https://mlsanity.com/paper/1909.05855/) (2019)
- [TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue](https://mlsanity.com/paper/2004.06871/) (2020)
- [Deep Voice 2: Multi-Speaker Neural Text-to-Speech](https://mlsanity.com/paper/1705.08947/) (2017)
- [DailyTalk: Spoken Dialogue Dataset for Conversational Text-to-Speech](https://mlsanity.com/paper/2207.01063/) (2022)
- [One TTS Alignment To Rule Them All](https://mlsanity.com/paper/2108.10447/) (2021)
- [Conversational End-to-End TTS for Voice Agent](https://mlsanity.com/paper/2005.10438/) (2020)
- [Duplex Conversation: Towards Human-like Interaction in Spoken Dialogue Systems](https://mlsanity.com/paper/2205.15060/) (2022)
- [MA-DST: Multi-Attention Based Scalable Dialog State Tracking](https://mlsanity.com/paper/2002.08898/) (2020)
- [Controllable Context-aware Conversational Speech Synthesis](https://mlsanity.com/paper/2106.10828/) (2021)
- [Modeling Long Context for Task-Oriented Dialogue State Generation](https://mlsanity.com/paper/2004.14080/) (2020)
- [Modeling Prosodic Phrasing with Multi-Task Learning in Tacotron-based TTS](https://mlsanity.com/paper/2008.05284/) (2020)
- [End-to-End Text-to-Speech Based on Latent Representation of Speaking Styles Using Spontaneous Dialogue](https://mlsanity.com/paper/2206.12040/) (2022)

## Built on by (most cited first, 9 of 9)

- [Generative Expressive Conversational Speech Synthesis](https://mlsanity.com/paper/2407.21491/) (2024)
- [CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matching](https://mlsanity.com/paper/2506.00885/) (2025)
- [CONCSS: Contrastive-based Context Comprehension for Dialogue-appropriate Prosody in Conversational Speech Synthesis](https://mlsanity.com/paper/2312.10358/) (2023)
- [Emphasis Rendering for Conversational Text-to-Speech with Multi-modal Multi-scale Context Modeling](https://mlsanity.com/paper/2410.09524/) (2024)
- [Retrieval-Augmented Dialogue Knowledge Aggregation for Expressive Conversational Speech Synthesis](https://mlsanity.com/paper/2501.06467/) (2025)
- [Towards Expressive Video Dubbing with Multiscale Multimodal Context Interaction](https://mlsanity.com/paper/2412.18748/) (2024)
- [Multimodal Fine-grained Context Interaction Graph Modeling for Conversational Speech Synthesis](https://mlsanity.com/paper/2509.06074/) (2025)
- [Let Me Look at You: Advanced Facial Expression Modeling for Conversational Speech Synthesis](https://mlsanity.com/paper/2607.24430/) (2026)
- [Alignment-Free Text-Audiobox for Voice Dubbing and Full-Duplex Dialogue Synthesis](https://mlsanity.com/paper/2609.03992/) (2026)

## Abstract (arXiv)

Conversational Text-to-Speech (TTS) aims to synthesis an utterance with the right linguistic and affective prosody in a conversational context. The correlation between the current utterance and the dialogue history at the utterance level was used to improve the expressiveness of synthesized speech. However, the fine-grained information in the dialogue history at the word level also has an important impact on the prosodic expression of an utterance, which has not been well studied in the prior work. Therefore, we propose a novel expressive conversational TTS model, termed as FCTalker, that learn the fine and coarse grained context dependency at the same time during speech generation. Specifically, the FCTalker includes fine and coarse grained encoders to exploit the word and utterance-level context dependency. To model the word-level dependencies between an utterance and its dialogue history, the fine-grained dialogue encoder is built on top of a dialogue BERT model. The experimental results show that the proposed method outperforms all baselines and generates more expressive speech that is contextually appropriate. We release the source code at: https://github.com/walker-hyf/FCTalker.