Is Prosody Lost in Translation? Fine-Grained Cross-Lingual Prosody Similarity Across Languages
Not on the map. Why?
Abstract (arXiv)
Prosody plays an important role in speech translation, conveying information such as emphasis, emotion, and intent beyond lexical content. However, despite recent progress in expressive speech-to-speech translation (S2ST), little is known about how prosodic patterns are similar/different across languages. Understanding these cross-lingual similarities and differences is crucial for effectively incorporating prosody into expressive S2ST systems. In this work, we present the first fine-grained cross-lingual analysis of prosody using multilingual dubbing data across English-German, English-Spanish, and English-French language pairs. We analyze the similarity of pitch, energy, and temporal feature patterns between source and target speech and investigate the linguistic and alignment-related factors affecting this similarity. Our analysis reveals inherent cross-lingual correlations in prosodic structure between certain languages. The findings provide important insights into the transferability of prosody across languages and offer empirical guidance for future expressive speech-to-speech translation systems.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 16 of its 36 references that are in the map. First posted on arXiv 28 Aug 2026.
Builds on
16 of 36 references.
References · 16
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
- 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 282
- 2019pyannote.audio: neural building blocks for speaker diarizationcited by 181
- 2021Direct speech-to-speech translation with discrete unitscited by 149
- 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
- 2023SONAR: Sentence-Level Multimodal and Language-Agnostic Representationscited by 108
- 2021Translatotron 2: High-quality direct speech-to-speech translation with voice preservationcited by 71
- 2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
- 2023TorchAudio-Squim: Reference-less Speech Quality and Intelligibility measures in TorchAudiocited by 59
- 2022Dubbing in Practice: A Large Scale Study of Human Localization With Insights for Automatic Dubbingcited by 17
- 2024Speech is More Than Words: Do Speech-to-Text Translation Systems Leverage Prosody?cited by 13
- 2023Towards cross-language prosody transfer for dialogcited by 5
4 more
- 2024Prosody in Cascade and Direct Speech-to-Text Translation: a case study on Korean Wh-Phrasescited by 5
- 2023Cross-lingual Prosody Transfer for Expressive Machine Dubbingcited by 4
- 2025Speech Vecalign: an Embedding-based Method for Aligning Parallel Speech Documentscited by 2
- 2025Improving French Synthetic Speech Quality via SSML Prosody Controlcited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.