mlsanityData to 14 Oct 2026

Speech is More Than Words: Do Speech-to-Text Translation Systems Leverage Prosody?

arXiv 2410.24019 ↗First posted 31 Oct 2024cs.CL cs.SD eess.AScited by 13

Not on the map. Why?

Abstract (arXiv)

The prosody of a spoken utterance, including features like stress, intonation and rhythm, can significantly affect the underlying semantics, and as a consequence can also affect its textual translation. Nevertheless, prosody is rarely studied within the context of speech-to-text translation (S2TT) systems. In particular, end-to-end (E2E) systems have been proposed as well-suited for prosody-aware translation because they have direct access to the speech signal when making translation decisions, but the understanding of whether this is successful in practice is still limited. A main challenge is the difficulty of evaluating prosody awareness in translation. To address this challenge, we introduce an evaluation methodology and a focused benchmark (named ContraProST) aimed at capturing a wide range of prosodic phenomena. Our methodology uses large language models and controllable text-to-speech (TTS) to generate contrastive examples. Through experiments in translating English speech into German, Spanish, and Japanese, we find that (a) S2TT models possess some internal representation of prosody, but the prosody signal is often not strong enough to affect the translations, (b) E2E systems outperform cascades of speech recognition and text translation systems, confirming their theoretical advantage in this regard, and (c) certain cascaded systems also capture prosodic information in the translation, but only to a lesser extent that depends on the particulars of the transcript's surface form.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 39 of its 79 references that are in the map. First posted on arXiv 31 Oct 2024.

Builds on

39 of 79 references.

References · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  8. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  9. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  10. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  11. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  12. 2022Large Language Models Can Self-Improvecited by 664
27 more
  1. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  2. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  3. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 388
  4. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  5. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  6. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 265
  7. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 250
  8. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 173
  9. 2022Prompting PaLM for Translation: Assessing Strategies and Performancecited by 145
  10. 2023Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventionscited by 125
  11. 2020Self-training and Pre-training are Complementary for Speech Recognitioncited by 99
  12. 2020Speech Translation and the End-to-End Promise: Taking Stock of Where We Arecited by 77
  13. 2020CTC-Segmentation of Large Corpora for German End-to-end Speech Recognitioncited by 76
  14. 2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
  15. 2023Steering Large Language Models for Machine Translation with Finetuning and In-Context Learningcited by 49
  16. 2016How Grammatical is Character-level Neural Machine Translation? Assessing MT Quality with Contrastive Translation Pairscited by 39
  17. 2023Speech Translation with Large Language Models: An Industrial Practicecited by 28
  18. 2023ProsAudit, a prosodic benchmark for self-supervised speech modelscited by 25
  19. 2021On the Limits of Minimal Pairs in Contrastive Evaluationcited by 15
  20. 2024Pushing the Limits of Zero-shot End-to-End Speech Translationcited by 14
  21. 2024Simpson's Paradox and the Accuracy-Fluency Tradeoff in Translationcited by 8
  22. 2021Prosody-TTS: An end-to-end speech synthesis system with prosody controlcited by 5
  23. 2021ADEPT: A Dataset for Evaluating Prosody Transfercited by 4
  24. 2024MELD-ST: An Emotion-aware Speech Translation Datasetcited by 4
  25. 2023Automating Behavioral Testing in Machine Translationcited by 2
  26. 2024Prosody for Intuitive Robotic Interface Design: It's Not What You Said, It's How You Said Itcited by 1
  27. 2024Conditioning LLMs with Emotion in Neural Machine Translationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 6.

13 papers.

Sort:

By area:

  1. 2025Hearing to Translate: The Effectiveness of Speech Modality Integration into LLMscited by 7
  2. 2025Speech-to-Text Translation with Phoneme-Augmented CoT: Enhancing Cross-Lingual Transfer in Low-Resource Scenarioscited by 3
  3. 2025Listening or Reading? Evaluating Speech Awareness in Chain-of-Thought Speech-to-Text Translationcited by 1
  4. 2026Why We Need Speech to Evaluate Speech Translationcited by 1
  5. 2026Is Prosody Lost in Translation? Fine-Grained Cross-Lingual Prosody Similarity Across Languagescited by 1
  6. 2025The Prosody of Emojiscited by 0
  7. 2025Toward Machine Interpreting: Lessons from Human Interpreting Studiescited by 0
  8. 2025Emphasis Sensitivity in Speech Representationscited by 0
  9. 2025Vision-Grounded Machine Interpreting: Improving the Translation Process through Visual Cuescited by 0
  10. 2026SimulU: Training-free Policy for Long-form Simultaneous Speech-to-Speech Translationcited by 0
  11. 2026Streaming Speech-to-Text Translation with a SpeechLLMcited by 0
  12. 2026Evaluating and Preserving Lexical Stress in English-to-Chinese Speech-to-Speech Translationcited by 0
1 more
  1. 2026In-Context Adaptation of Encoder-Decoder Models in Speech Recognitioncited by 0