# CONCSS: Contrastive-based Context Comprehension for Dialogue-appropriate Prosody in Conversational Speech Synthesis

arXiv: https://arxiv.org/abs/2312.10358 · HTML: https://mlsanity.com/paper/2312.10358/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 23 references that are in the map.
- First posted on arXiv 16 Dec 2023.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers](https://mlsanity.com/paper/2304.09116/) (2023)
- [DailyTalk: Spoken Dialogue Dataset for Conversational Text-to-Speech](https://mlsanity.com/paper/2207.01063/) (2022)
- [Conversational End-to-End TTS for Voice Agent](https://mlsanity.com/paper/2005.10438/) (2020)
- [ProsoSpeech: Enhancing Prosody With Quantized Vector Pre-training in Text-to-Speech](https://mlsanity.com/paper/2202.07816/) (2022)
- [Enhancing Speaking Styles in Conversational Text-to-Speech Synthesis with Graph-based Multi-modal Context Modeling](https://mlsanity.com/paper/2106.06233/) (2021)
- [FCTalker: Fine and Coarse Grained Context Modeling for Expressive Conversational Speech Synthesis](https://mlsanity.com/paper/2210.15360/) (2022)
- [M2-CTTS: End-to-End Multi-scale Multi-modal Conversational Text-to-Speech Synthesis](https://mlsanity.com/paper/2305.02269/) (2023)
- [Towards Multi-Scale Speaking Style Modelling with Hierarchical Context Information for Mandarin Speech Synthesis](https://mlsanity.com/paper/2204.02743/) (2022)
- [Acoustic Modeling for End-to-End Empathetic Dialogue Speech Synthesis Using Linguistic and Prosodic Contexts of Dialogue History](https://mlsanity.com/paper/2206.08039/) (2022)

## Built on by (most cited first, 8 of 8)

- [Generative Expressive Conversational Speech Synthesis](https://mlsanity.com/paper/2407.21491/) (2024)
- [Intra- and Inter-modal Context Interaction Modeling for Conversational Speech Synthesis](https://mlsanity.com/paper/2412.18733/) (2024)
- [Retrieval-Augmented Dialogue Knowledge Aggregation for Expressive Conversational Speech Synthesis](https://mlsanity.com/paper/2501.06467/) (2025)
- [Improving Audio Codec-based Zero-Shot Text-to-Speech Synthesis with Multi-Modal Context and Large Language Model](https://mlsanity.com/paper/2406.03706/) (2024)
- [Which Prosodic Features Matter Most for Pragmatics?](https://mlsanity.com/paper/2408.13240/) (2024)
- [DiffCSS: Diverse and Expressive Conversational Speech Synthesis with Diffusion Models](https://mlsanity.com/paper/2502.19924/) (2025)
- [Multimodal Fine-grained Context Interaction Graph Modeling for Conversational Speech Synthesis](https://mlsanity.com/paper/2509.06074/) (2025)
- [AuEmoChat: Authentic Emotion Understanding and Rendering for Conversational Speech Synthesis](https://mlsanity.com/paper/2607.15755/) (2026)

## Abstract (arXiv)

Conversational speech synthesis (CSS) incorporates historical dialogue as supplementary information with the aim of generating speech that has dialogue-appropriate prosody. While previous methods have already delved into enhancing context comprehension, context representation still lacks effective representation capabilities and context-sensitive discriminability. In this paper, we introduce a contrastive learning-based CSS framework, CONCSS. Within this framework, we define an innovative pretext task specific to CSS that enables the model to perform self-supervised learning on unlabeled conversational datasets to boost the model's context understanding. Additionally, we introduce a sampling strategy for negative sample augmentation to enhance context vectors' discriminability. This is the first attempt to integrate contrastive learning into CSS. We conduct ablation studies on different contrastive learning strategies and comprehensive experiments in comparison with prior CSS systems. Results demonstrate that the synthesized speech from our proposed method exhibits more contextually appropriate and sensitive prosody.