mlsanityData to 14 Oct 2026

Conditioned in Training, Dispensable at Inference: Typological Conditioning for Multilingual Speech Translation

arXiv 2605.16026 ↗First posted 15 May 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Multilingual speech translation frontends built on speech large language models typically condition on language identity, as a learned embedding or a prompted language name. Such a signal, however, is learned separately for each language and transfers nothing between typologically related languages. In this paper, we present TypoCast, a frontend that conditions on the typological profile of the source language rather than on its label alone. A typology-informed hierarchical language encoding represents each language by its morphological type, constituent order and family, so that supervision is shared among related languages. A dual-route mixture-of-experts adapter routes inputs through a shared typological route together with a per-language residual, and a typology-aware prompt conveys the same profile to the decoder. Extensive experiments on CVSS-C demonstrate that TypoCast surpasses state-of-the-art speech-to-speech translation systems on French, German and Spanish to English, averaged over the three directions, and outperforms the identity-conditioned S2ST-Omni on every language of a seven-language inventory, with only 228M trainable parameters. Ablation studies further reveal that the conditioning signal acts during training and is dispensable at inference: a model trained without it is consistently worse, whereas removing it from a trained model changes BLEU by less than 0.2, and expert usage is consistent with the signal shaping how adapter capacity is divided among languages.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 36 of its 49 references that are in the map. First posted on arXiv 15 May 2026.

Builds on

36 of 49 references.

References · 36

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  6. 2023Qwen Technical Reportcited by 3,344
  7. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  8. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  9. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  10. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  11. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
  12. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 265
24 more
  1. 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-trainingcited by 150
  2. 2023Towards Making the Most of ChatGPT for Machine Translationcited by 138
  3. 2023SeamlessM4T: Massively Multilingual & Multimodal Machine Translationcited by 125
  4. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  5. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  6. 2025IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speechcited by 90
  7. 2021Translatotron 2: High-quality direct speech-to-speech translation with voice preservationcited by 71
  8. 2022CVSS Corpus and Massively Multilingual Speech-to-Speech Translationcited by 71
  9. 2018Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processingcited by 63
  10. 2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
  11. 2023Adaptive Machine Translation with Large Language Modelscited by 58
  12. 2025VoxCPM: Tokenizer-Free TTS for Context-Aware Speech Generation and True-to-Life Voice Cloningcited by 42
  13. 2025FireRedTTS-2: Towards Long Conversational Speech Generation for Podcast and Chatbotcited by 35
  14. 2025ZipVoice: Fast and High-Quality Zero-Shot Text-to-Speech with Flow Matchingcited by 30
  15. 2020Bridging Linguistic Typology and Multilingual Machine Translation with Multi-View Language Representationscited by 21
  16. 2025High-Fidelity Simultaneous Speech-To-Speech Translationcited by 15
  17. 2024StableVC: Style Controllable Zero-Shot Voice Conversion with Conditional Flow Matchingcited by 14
  18. 2023HYBRIDFORMER: improving SqueezeFormer with hybrid attention and NSR mechanismcited by 11
  19. 2024Analyzing the Evaluation of Cross-Lingual Knowledge Transfer in Multilingual Language Modelscited by 11
  20. 2023DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translationcited by 9
  21. 2024Takin: A Cohort of Superior Quality Zero-shot Speech Generation Modelscited by 7
  22. 2024Analysis of Multi-Source Language Training in Cross-Lingual Transfercited by 5
  23. 2024Can We Achieve High-quality Direct Speech-to-Speech Translation without Parallel Speech Data?cited by 4
  24. 2025S2ST-Omni: Hierarchical Language-Aware SpeechLLM Adaptation for Multilingual Speech-to-Speech Translationcited by 1

Built on by

Not cited yet.