# Is Prosody Lost in Translation? Fine-Grained Cross-Lingual Prosody Similarity Across Languages

arXiv: https://arxiv.org/abs/2608.27848 · HTML: https://mlsanity.com/paper/2608.27848/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 36 references that are in the map.
- First posted on arXiv 28 Aug 2026.

## Builds on (most cited first, 16 of 16)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](https://mlsanity.com/paper/1806.04558/) (2018)
- [pyannote.audio: neural building blocks for speaker diarization](https://mlsanity.com/paper/1911.01255/) (2019)
- [Direct speech-to-speech translation with discrete units](https://mlsanity.com/paper/2107.05604/) (2021)
- [Direct speech-to-speech translation with a sequence-to-sequence model](https://mlsanity.com/paper/1904.06037/) (2019)
- [SONAR: Sentence-Level Multimodal and Language-Agnostic Representations](https://mlsanity.com/paper/2308.11466/) (2023)
- [Translatotron 2: High-quality direct speech-to-speech translation with voice preservation](https://mlsanity.com/paper/2107.08661/) (2021)
- [UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Units](https://mlsanity.com/paper/2212.08055/) (2022)
- [TorchAudio-Squim: Reference-less Speech Quality and Intelligibility measures in TorchAudio](https://mlsanity.com/paper/2304.01448/) (2023)
- [Dubbing in Practice: A Large Scale Study of Human Localization With Insights for Automatic Dubbing](https://mlsanity.com/paper/2212.12137/) (2022)
- [Speech is More Than Words: Do Speech-to-Text Translation Systems Leverage Prosody?](https://mlsanity.com/paper/2410.24019/) (2024)
- [Towards cross-language prosody transfer for dialog](https://mlsanity.com/paper/2307.04123/) (2023)
- [Prosody in Cascade and Direct Speech-to-Text Translation: a case study on Korean Wh-Phrases](https://mlsanity.com/paper/2402.00632/) (2024)
- [Cross-lingual Prosody Transfer for Expressive Machine Dubbing](https://mlsanity.com/paper/2306.11658/) (2023)
- [Speech Vecalign: an Embedding-based Method for Aligning Parallel Speech Documents](https://mlsanity.com/paper/2509.18360/) (2025)
- [Improving French Synthetic Speech Quality via SSML Prosody Control](https://mlsanity.com/paper/2508.17494/) (2025)

## Built on by (most cited first, 1 of 1)

- [All In Good Time: Causality-Aware Framework for LLM-Based Simultaneous Speech-to-Speech Translation](https://mlsanity.com/paper/2609.30416/) (2026)

## Abstract (arXiv)

Prosody plays an important role in speech translation, conveying information such as emphasis, emotion, and intent beyond lexical content. However, despite recent progress in expressive speech-to-speech translation (S2ST), little is known about how prosodic patterns are similar/different across languages. Understanding these cross-lingual similarities and differences is crucial for effectively incorporating prosody into expressive S2ST systems. In this work, we present the first fine-grained cross-lingual analysis of prosody using multilingual dubbing data across English-German, English-Spanish, and English-French language pairs. We analyze the similarity of pitch, energy, and temporal feature patterns between source and target speech and investigate the linguistic and alignment-related factors affecting this similarity. Our analysis reveals inherent cross-lingual correlations in prosodic structure between certain languages. The findings provide important insights into the transferability of prosody across languages and offer empirical guidance for future expressive speech-to-speech translation systems.