mlsanityData to 14 Oct 2026

Multi-Dimensional Prosody Judgment For Live Streaming Speech Synthesis

arXiv 2609.20124 ↗First posted 17 Sep 2026cs.SD cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Evaluating live streaming speech synthesis (TTS) requires assessing fine-grained, highly expressive prosody such as emotion, intonation, and energy which traditional MOS predictors fail to capture. While proprietary Large Language Models (LLMs) like Gemini can evaluate these aspects, they are too costly for massive inference and reinforcement learning feedback. To address this, we first introduce Live-ProsodyJudge (LPJ), a cost-effective pairwise evaluator distilled from Gemini into Qwen3-Omni. However, we identify a critical flaw in standard multi-dimensional evaluation: verdict coupling. The judge tends to lazily align all individual dimension scores with its overall preference, collapsing a rich multi-dimensional rubric into a single preference bit. To resolve this, we further propose Decoupled-Live-ProsodyJudge (D-LPJ). D-LPJ eliminates the overall verdict target to prevent blind following, masks uncertain pair-dimensions during Supervised Fine-Tuning(SFT), and introduces a novel span-local GRPO strategy that applies normalized advantages strictly to their corresponding rationale spans. Evaluated on highly curated human-annotated test sets, 10 sample balanced-order LPJ achieves higher point accuracy than a single Gemini call, while D-LPJ successfully produces independent,decoupled dimension judgments. Furthermore, in a Best-of-8 TTS candidate selection tournament, the LPJ-selected utterance falls within the human top-3 in 85.29% of high-confidence cases, demonstrating its efficacy for fine-grained TTS preference optimization.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 18 of its 19 references that are in the map. First posted on arXiv 17 Sep 2026.

Builds on

18 of 19 references.

References · 18

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3-Omni Technical Reportcited by 497
  4. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 390
  5. 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 355
  6. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
  7. 2021NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasetscited by 192
  8. 2019MOSNet: Deep Learning based Objective Assessment for Voice Conversioncited by 127
  9. 2026Qwen3-TTS Technical Reportcited by 118
  10. 2025IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speechcited by 90
  11. 2024FireRedTTS: A Foundation Text-To-Speech Framework for Industry-Level Generative Speech Applicationscited by 68
  12. 2025EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judgecited by 24
6 more
  1. 2025SpeechJudge: Towards Human-Level Judgment for Speech Naturalnesscited by 15
  2. 2026GSRM: Generative Speech Reward Model for Speech RLHFcited by 4
  3. 2026dots.tts Technical Reportcited by 4
  4. 2026Evaluating the Expressive Appropriateness of Speech in Rich Contextscited by 3
  5. 2026UniSRM: A Unified Speech Reward Model for Reasoning-Based Fine-grained Assessmentcited by 2
  6. 2026Comprehensive Benchmarking of Long-Form Speech Generation in Diverse Scenarioscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Expressive Zero-Shot Speech Synthesis with Hierarchical Prosody Modeling
  2. 2025Investigating Stochastic Methods for Prosody Modeling in Speech Synthesis
  3. 2024ProsodyFM: Unsupervised Phrasing and Intonation Control for Intelligible Speech Synthesis
  4. 2023Diverse and Expressive Speech Prosody Prediction with Denoising Diffusion Probabilistic Model
  5. 2025RLAIF-SPA: Structured AI Feedback for Semantic-Prosodic Alignment in Speech Synthesis
  6. 2025No Verifiable Reward for Prosody: Toward Preference-Guided Prosody Learning in TTS
  7. 2026CuteTTS: Efficient and High-Quality Speech Synthesis via Autoregressive Modeling of Continuous Latents
  8. 2023DiffProsody: Diffusion-based Latent Prosody Generation for Expressive Speech Synthesis with Prosody Conditional Adversarial Training