mlsanityData to 7 Oct 2026

Cross-Lingual Transfer Learning for Speech Translation

arXiv 2407.01130 ↗First posted 1 Jul 2024cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

There has been increasing interest in building multilingual foundation models for NLP and speech research. This paper examines how to expand the speech translation capability of these models with restricted data. Whisper, a speech foundation model with strong performance on speech recognition and English translation, is used as the example model. Using speech-to-speech retrieval to analyse the audio representations generated by the encoder, we show that utterances from different languages are mapped to a shared semantic space. This shared embedding space can then be leveraged for zero-shot cross-lingual transfer in speech translation. By fine-tuning the Whisper decoder with only English-to-Chinese speech translation data, improved performance for translation to Chinese can be obtained for multiple languages, in addition to English. Furthermore, for languages related to those seen in training it is possible to perform speech translation, despite the model never seeing the language in training, or being able to perform transcription.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 35 of its 48 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

35 of 48 references.

References · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  7. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  9. 2018Deep contextualized word representationscited by 2,285
  10. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  11. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  12. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
23 more
  1. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  2. 2019How multilingual is Multilingual BERT?cited by 642
  3. 2020Language-agnostic BERT Sentence Embeddingcited by 621
  4. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  5. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  6. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  7. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
  8. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  9. 2023Seamless: Multilingual Expressive and Streaming Speech Translationcited by 190
  10. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 155
  11. 2020Multilingual Alignment of Contextual Word Representationscited by 113
  12. 2023SONAR: Sentence-Level Multimodal and Language-Agnostic Representationscited by 105
  13. 2021Larger-Scale Transformers for Multilingual Masked Language Modelingcited by 86
  14. 2018Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Inputcited by 81
  15. 2017Learning Joint Multilingual Sentence Representations with Neural Machine Translationcited by 52
  16. 2023Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalizationcited by 43
  17. 2019Multilingual End-to-End Speech Translationcited by 37
  18. 2022SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representationcited by 32
  19. 2022Probing phoneme, language and speaker information in unsupervised speech representationscited by 31
  20. 2019Fine-grained robust prosody transfer for single-speaker neural text-to-speechcited by 27
  21. 2023Pre-training for Speech Translation: CTC Meets Optimal Transportcited by 25
  22. 2019MaSS: A Large and Clean Multilingual Corpus of Sentence-aligned Spoken Utterances Extracted from the Biblecited by 24
  23. 2023Investigating the Emergent Audio Classification Ability of ASR Foundation Modelscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 6.

11 papers.

  1. 2026PRiSM: Benchmarking Phone Realization in Speech Modelscited by 4
  2. 2025Languages in Whisper-Style Speech Encoders Align Both Phonetically and Semanticallycited by 3
  3. 2025Scaling and Prompting for Improved End-to-End Spoken Grammatical Error Correctioncited by 2
  4. 2025End-to-End Spoken Grammatical Error Correctioncited by 1
  5. 2026Multimodal In-context Learning for ASR of Low-resource Languagescited by 1
  6. 2026BiST: A Gold Standard Bangla-English Bilingual Corpus for Sentence Structure and Tense Classification with Inter-Annotator Agreementcited by 1
  7. 2025Whisper-UT: A Unified Translation Framework for Speech and Textcited by 0
  8. 2025POTSA: A Cross-Lingual Speech Alignment Framework for Speech-to-Text Translationcited by 0
  9. 2026Linear Script Representations in Speech Foundation Models Enable Zero-Shot Transliterationcited by 0
  10. 2026Towards Comprehensive Semantic Speech Embeddings for Chinese Dialectscited by 0
  11. 2026Does Translation-Enhanced Speech Encoder Pre-training Affect Speech LLMs?cited by 0