mlsanityData to 7 Oct 2026

M2-CTTS: End-to-End Multi-scale Multi-modal Conversational Text-to-Speech Synthesis

arXiv 2305.02269 ↗First posted 3 May 2023cs.SD cs.CL eess.AScited by 6

Not on the map. Why?

Abstract (arXiv)

Conversational text-to-speech (TTS) aims to synthesize speech with proper prosody of reply based on the historical conversation. However, it is still a challenge to comprehensively model the conversation, and a majority of conversational TTS systems only focus on extracting global information and omit local prosody features, which contain important fine-grained information like keywords and emphasis. Moreover, it is insufficient to only consider the textual features, and acoustic features also contain various prosody information. Hence, we propose M2-CTTS, an end-to-end multi-scale multi-modal conversational text-to-speech system, aiming to comprehensively utilize historical conversation and enhance prosodic expression. More specifically, we design a textual context module and an acoustic context module with both coarse-grained and fine-grained modeling. Experimental results demonstrate that our model mixed with fine-grained context information and additionally considering acoustic features achieves better prosody performance and naturalness in CMOS tests.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 17 of its 21 references that are in the map. First posted on arXiv 3 May 2023.

Builds on

17 of 21 references.

References · 17

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  4. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  5. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  6. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 649
  7. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 257
  8. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 171
  9. 2021Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generationcited by 93
  10. 2022DailyTalk: Spoken Dialogue Dataset for Conversational Text-to-Speechcited by 67
  11. 2018Robust and fine-grained prosody control of end-to-end speech synthesiscited by 48
  12. 2021One TTS Alignment To Rule Them Allcited by 47
5 more
  1. 2020Conversational End-to-End TTS for Voice Agentcited by 37
  2. 2022MsEmoTTS: Multi-scale emotion transfer, prediction, and control for emotional speech synthesiscited by 29
  3. 2021Controllable Context-aware Conversational Speech Synthesiscited by 18
  4. 2021Enhancing Speaking Styles in Conversational Text-to-Speech Synthesis with Graph-based Multi-modal Context Modelingcited by 15
  5. 2021Cross-speaker Emotion Transfer Based on Speaker Condition Layer Normalization and Semi-Supervised Training in Text-To-Speechcited by 8

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 2; 2025: 1.

6 papers.

  1. 2023StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Modelscited by 148
  2. 2023Towards human-like spoken dialogue generation between AI agents from written dialoguecited by 19
  3. 2025CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matchingcited by 11
  4. 2023CONCSS: Contrastive-based Context Comprehension for Dialogue-appropriate Prosody in Conversational Speech Synthesiscited by 8
  5. 2024Pheme: Efficient and Conversational Speech Generationcited by 6
  6. 2024Improving Audio Codec-based Zero-Shot Text-to-Speech Synthesis with Multi-Modal Context and Large Language Modelcited by 3