# MoVE: Translating Laughter and Tears via Mixture of Vocalization Experts in Speech-to-Speech Translation

arXiv: https://arxiv.org/abs/2604.17435 · HTML: https://mlsanity.com/paper/2604.17435/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 36 references that are in the map.
- First posted on arXiv 19 Apr 2026.

## Builds on (most cited first, 26 of 26)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio](https://mlsanity.com/paper/2106.06909/) (2021)
- [Kimi-Audio Technical Report](https://mlsanity.com/paper/2504.18425/) (2025)
- [Seamless: Multilingual Expressive and Streaming Speech Translation](https://mlsanity.com/paper/2312.05187/) (2023)
- [Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling](https://mlsanity.com/paper/2303.03926/) (2023)
- [Dawn of the transformer era in speech emotion recognition: closing the valence gap](https://mlsanity.com/paper/2203.07378/) (2022)
- [CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training](https://mlsanity.com/paper/2505.17589/) (2025)
- [Direct speech-to-speech translation with discrete units](https://mlsanity.com/paper/2107.05604/) (2021)
- [SeamlessM4T: Massively Multilingual & Multimodal Machine Translation](https://mlsanity.com/paper/2308.11596/) (2023)
- [On The Landscape of Spoken Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2504.08528/) (2025)
- [IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speech](https://mlsanity.com/paper/2506.21619/) (2025)
- [Translatotron 2: High-quality direct speech-to-speech translation with voice preservation](https://mlsanity.com/paper/2107.08661/) (2021)
- [CVSS Corpus and Massively Multilingual Speech-to-Speech Translation](https://mlsanity.com/paper/2201.03713/) (2022)
- [X-LoRA: Mixture of Low-Rank Adapter Experts, a Flexible Framework for Large Language Models with Applications in Protein Mechanics and Molecular Design](https://mlsanity.com/paper/2402.07148/) (2024)
- [Towards audio language modeling -- an overview](https://mlsanity.com/paper/2402.13236/) (2024)
- [GigaST: A 10,000-hour Pseudo Speech Translation Corpus](https://mlsanity.com/paper/2204.03939/) (2022)
- [Laugh Now Cry Later: Controlling Time-Varying Emotional States of Flow-Matching-Based Zero-Shot Text-to-Speech](https://mlsanity.com/paper/2407.12229/) (2024)
- [NonverbalTTS: A Public English Corpus of Text-Aligned Nonverbal Vocalizations with Emotion Annotations for Text-to-Speech](https://mlsanity.com/paper/2507.13155/) (2025)
- [Translatotron 3: Speech to Speech Translation with Monolingual Data](https://mlsanity.com/paper/2305.17547/) (2023)
- [JVNV: A Corpus of Japanese Emotional Speech with Verbal Content and Nonverbal Expressions](https://mlsanity.com/paper/2310.06072/) (2023)
- [SeamlessExpressiveLM: Speech Language Model for Expressive Speech-to-Speech Translation with Chain-of-Thought](https://mlsanity.com/paper/2405.20410/) (2024)
- [Towards cross-language prosody transfer for dialog](https://mlsanity.com/paper/2307.04123/) (2023)

## Abstract (arXiv)

Recent Speech-to-Speech Translation (S2ST) systems achieve strong semantic accuracy yet consistently strip away non-verbal vocalizations (NVs), such as laughter and crying that convey pragmatic intent, which severely limits real-world utility. We address this via three contributions. First, we propose a synthesis pipeline for building scalable expressive datasets to overcome the data scarcity limitation. Second, we propose MoVE, a Mixture-of-LoRA-Experts architecture with expressive-specialized adapters and a soft-weighting router that blends experts for capturing hybrid expressive states. Third, we show pretrained AudioLLMs enable striking data efficiency: 30 minutes of curated data is enough for strong performance. On English-Chinese S2ST, while comparing with strong baselines, MoVE reproduces target NVs in 76% of cases and achieves the highest human-rated naturalness and emotional fidelity among all compared systems, where existing S2ST systems preserve at most 14% of NVs.