mlsanityData to 14 Oct 2026

MoVE: Translating Laughter and Tears via Mixture of Vocalization Experts in Speech-to-Speech Translation

arXiv 2604.17435 ↗First posted 19 Apr 2026cs.CL cs.AI cs.SDcited by 0

Not on the map. Why?

Abstract (arXiv)

Recent Speech-to-Speech Translation (S2ST) systems achieve strong semantic accuracy yet consistently strip away non-verbal vocalizations (NVs), such as laughter and crying that convey pragmatic intent, which severely limits real-world utility. We address this via three contributions. First, we propose a synthesis pipeline for building scalable expressive datasets to overcome the data scarcity limitation. Second, we propose MoVE, a Mixture-of-LoRA-Experts architecture with expressive-specialized adapters and a soft-weighting router that blends experts for capturing hybrid expressive states. Third, we show pretrained AudioLLMs enable striking data efficiency: 30 minutes of curated data is enough for strong performance. On English-Chinese S2ST, while comparing with strong baselines, MoVE reproduces target NVs in 76% of cases and achieves the highest human-rated naturalness and emotional fidelity among all compared systems, where existing S2ST systems preserve at most 14% of NVs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 26 of its 36 references that are in the map. First posted on arXiv 19 Apr 2026.

Builds on

26 of 36 references.

References · 26

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2024GPT-4o System Cardcited by 4,588
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  4. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  5. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  6. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 399
  7. 2025Kimi-Audio Technical Reportcited by 222
  8. 2023Seamless: Multilingual Expressive and Streaming Speech Translationcited by 192
  9. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 167
  10. 2022Dawn of the transformer era in speech emotion recognition: closing the valence gapcited by 157
  11. 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-trainingcited by 150
  12. 2021Direct speech-to-speech translation with discrete unitscited by 149
14 more
  1. 2023SeamlessM4T: Massively Multilingual & Multimodal Machine Translationcited by 125
  2. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  3. 2025IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speechcited by 90
  4. 2021Translatotron 2: High-quality direct speech-to-speech translation with voice preservationcited by 71
  5. 2022CVSS Corpus and Massively Multilingual Speech-to-Speech Translationcited by 71
  6. 2024X-LoRA: Mixture of Low-Rank Adapter Experts, a Flexible Framework for Large Language Models with Applications in Protein Mechanics and Molecular Designcited by 42
  7. 2024Towards audio language modeling -- an overviewcited by 40
  8. 2022GigaST: A 10,000-hour Pseudo Speech Translation Corpuscited by 20
  9. 2024Laugh Now Cry Later: Controlling Time-Varying Emotional States of Flow-Matching-Based Zero-Shot Text-to-Speechcited by 19
  10. 2025NonverbalTTS: A Public English Corpus of Text-Aligned Nonverbal Vocalizations with Emotion Annotations for Text-to-Speechcited by 14
  11. 2023Translatotron 3: Speech to Speech Translation with Monolingual Datacited by 12
  12. 2023JVNV: A Corpus of Japanese Emotional Speech with Verbal Content and Nonverbal Expressionscited by 10
  13. 2024SeamlessExpressiveLM: Speech Language Model for Expressive Speech-to-Speech Translation with Chain-of-Thoughtcited by 9
  14. 2023Towards cross-language prosody transfer for dialogcited by 5

Built on by

Not cited yet.