mlsanityData to 14 Oct 2026

Optimizing Multilingual Text-To-Speech with Accents & Emotions

arXiv 2506.16310 ↗First posted 19 Jun 2025cs.LG cs.HC cs.SDcited by 0

Not on the map. Why?

Abstract (arXiv)

State-of-the-art text-to-speech (TTS) systems realize high naturalness in monolingual environments, synthesizing speech with correct multilingual accents (especially for Indic languages) and context-relevant emotions still poses difficulty owing to cultural nuance discrepancies in current frameworks. This paper introduces a new TTS architecture integrating accent along with preserving transliteration with multi-scale emotion modelling, in particularly tuned for Hindi and Indian English accent. Our approach extends the Parler-TTS model by integrating A language-specific phoneme alignment hybrid encoder-decoder architecture, and culture-sensitive emotion embedding layers trained on native speaker corpora, as well as incorporating a dynamic accent code switching with residual vector quantization. Quantitative tests demonstrate 23.7% improvement in accent accuracy (Word Error Rate reduction from 15.4% to 11.8%) and 85.3% emotion recognition accuracy from native listeners, surpassing METTS and VECL-TTS baselines. The novelty of the system is that it can mix code in real time - generating statements such as "Namaste, let's talk about <Hindi phrase>" with uninterrupted accent shifts while preserving emotional consistency. Subjective evaluation with 200 users reported a mean opinion score (MOS) of 4.2/5 for cultural correctness, much better than existing multilingual systems (p<0.01). This research makes cross-lingual synthesis more feasible by showcasing scalable accent-emotion disentanglement, with direct application in South Asian EdTech and accessibility software.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 34 references that are in the map. First posted on arXiv 19 Jun 2025.

Builds on

22 of 34 references.

References · 22

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  4. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  5. 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 282
  6. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generationcited by 257
  7. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 205
  8. 2025Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokenscited by 113
  9. 2024Natural language guidance of high-fidelity text-to-speech with synthetic annotationscited by 67
  10. 2019Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloningcited by 66
  11. 2025EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Promptingcited by 34
  12. 2024IndicVoices-R: Unlocking a Massive Multilingual Multi-speaker Speech Corpus for Scaling Indian TTScited by 16
10 more
  1. 2023Accented Text-to-Speech Synthesis with Limited Datacited by 12
  2. 2024ED-TTS: Multi-Scale Emotion Modeling using Cross-Domain Emotion Diarization for Emotional Speech Synthesiscited by 9
  3. 2023METTS: Multilingual Emotional Text-to-Speech by Cross-speaker and Cross-lingual Emotion Transfercited by 6
  4. 2022Explicit Intensity Control for Accented Text-to-speechcited by 5
  5. 2023Boosting Fast and High-Quality Speech Synthesis with Linear Diffusioncited by 4
  6. 2024VECL-TTS: Voice identity and Emotional style controllable Cross-Lingual Text-to-Speechcited by 4
  7. 2023Multilingual Text-to-Speech Synthesis for Turkic Languages Using Transliterationcited by 2
  8. 2023DiCLET-TTS: Diffusion Model based Cross-lingual Emotion Transfer for Text-to-Speech -- A Study between English and Mandarincited by 2
  9. 2024VQalAttent: a Transparent Speech Generation Pipeline based on Transformer-learned VQ-VAE Latent Spacecited by 2
  10. 2022Detection of AI Synthesized Hindi Speechcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023AffectEcho: Speaker Independent and Language-Agnostic Emotion and Affect Transfer for Speech Synthesis
  2. 2025LatinX: Aligning a Multilingual TTS Model with Direct Preference Optimization
  3. 2025MERaLiON-SER: Robust Speech Emotion Recognition Model for English and SEA Languages
  4. 2024DART: Disentanglement of Accent and Speaker Representation in Multispeaker Text-to-Speech
  5. 2025CLARITY: Contextual Linguistic Adaptation and Accent Retrieval for Dual-Bias Mitigation in Text-to-Speech Generation
  6. 2025Developing multilingual speech synthesis system for Ojibwe, Mi'kmaq, and Maliseet
  7. 2025ALAS: An Automatic Latent Alignment Score for Audio Language Models
  8. 2026Cross-Lingual Sentiment Misalignment: Auditing Multilingual Language Models for Inversion Risk, Dialectal Representation, and Affective Stability