# Evaluating and Preserving Lexical Stress in English-to-Chinese Speech-to-Speech Translation

arXiv: https://arxiv.org/abs/2606.15266 · HTML: https://mlsanity.com/paper/2606.15266/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 30 references that are in the map.
- First posted on arXiv 13 Jun 2026.

## Builds on (most cited first, 21 of 21)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022](https://mlsanity.com/paper/2204.02152/) (2022)
- [Layer-wise Analysis of a Self-supervised Speech Representation Model](https://mlsanity.com/paper/2107.04734/) (2021)
- [Seamless: Multilingual Expressive and Streaming Speech Translation](https://mlsanity.com/paper/2312.05187/) (2023)
- [SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddings](https://mlsanity.com/paper/2004.08728/) (2020)
- [CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training](https://mlsanity.com/paper/2505.17589/) (2025)
- [SeamlessM4T: Massively Multilingual & Multimodal Machine Translation](https://mlsanity.com/paper/2308.11596/) (2023)
- [Direct speech-to-speech translation with a sequence-to-sequence model](https://mlsanity.com/paper/1904.06037/) (2019)
- [UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Units](https://mlsanity.com/paper/2212.08055/) (2022)
- [Speech-to-Speech Translation For A Real-world Unwritten Language](https://mlsanity.com/paper/2211.06474/) (2022)
- [EmphAssess : a Prosodic Benchmark on Assessing Emphasis Transfer in Speech-to-Speech Models](https://mlsanity.com/paper/2312.14069/) (2023)
- [PolyVoice: Language Models for Speech to Speech Translation](https://mlsanity.com/paper/2306.02982/) (2023)
- [BLASER: A Text-Free Speech-to-Speech Translation Evaluation Metric](https://mlsanity.com/paper/2212.08486/) (2022)
- [Speech is More Than Words: Do Speech-to-Text Translation Systems Leverage Prosody?](https://mlsanity.com/paper/2410.24019/) (2024)
- [StressTest: Can YOUR Speech LM Handle the Stress?](https://mlsanity.com/paper/2505.22765/) (2025)
- [Direct Speech-to-Speech Neural Machine Translation: A Survey](https://mlsanity.com/paper/2411.14453/) (2024)
- [Direct Speech to Speech Translation: A Review](https://mlsanity.com/paper/2503.04799/) (2025)
- [Attempt Towards Stress Transfer in Speech-to-Speech Machine Translation](https://mlsanity.com/paper/2403.04178/) (2024)
- [StressTransfer: Stress-Aware Speech-to-Speech Translation with Emphasis Preservation](https://mlsanity.com/paper/2510.13194/) (2025)

## Abstract (arXiv)

Speech-to-speech translation (S2ST) systems have achieved impressive progress in semantic accuracy and speech naturalness. However, the cross-lingual transfer of lexical stress, a vital cue for emphasis and speaker intent, remains heavily underexplored, compounded by a lack of reliable automatic evaluation metrics for tonal languages like Chinese. We investigate English-to-Chinese S2ST stress transfer by constructing a stress-annotated Chinese dataset and an XLS-R-based Mandarin stress detector. Integrating this with the English EmphAssess system, we propose a novel objective metric for cross-lingual stress evaluation. Furthermore, we fine-tune CosyVoice3 to build a stress-aware S2ST system. Experiments demonstrate that our proposed S2ST architecture significantly outperforms existing systems in stress translation capability while maintaining competitive translation quality. Furthermore, our evaluation metric exhibits a strong correlation with human subjective judgments.