mlsanityData to 7 Oct 2026

Pushing the Limits of Zero-shot End-to-End Speech Translation

arXiv 2402.10422 ↗First posted 16 Feb 2024cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

Data scarcity and the modality gap between the speech and text modalities are two major obstacles of end-to-end Speech Translation (ST) systems, thus hindering their performance. Prior work has attempted to mitigate these challenges by leveraging external MT data and optimizing distance metrics that bring closer the speech-text representations. However, achieving competitive results typically requires some ST data. For this reason, we introduce ZeroSwot, a method for zero-shot ST that bridges the modality gap without any paired ST data. Leveraging a novel CTC compression and Optimal Transport, we train a speech encoder using only ASR data, to align with the representation space of a massively multilingual MT model. The speech encoder seamlessly integrates with the MT model at inference, enabling direct translation from speech to text, across all languages supported by the MT model. Our experiments show that we can effectively close the modality gap without ST data, while our results on MuST-C and CoVoST demonstrate our method's superiority over not only previous zero-shot models, but also supervised ones, achieving state-of-the-art results.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 57 of its 83 references that are in the map. First posted on arXiv 16 Feb 2024.

Builds on

57 of 83 references.

References · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  6. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  8. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  9. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  10. 2020On Layer Normalization in the Transformer Architecturecited by 906
  11. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  12. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
38 more
  1. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  2. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  3. 2015Learning with a Wasserstein Losscited by 194
  4. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 174
  5. 2020Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metricscited by 155
  6. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
  7. 2020Self-training and Pre-training are Complementary for Speech Recognitioncited by 98
  8. 2020Speech Translation and the End-to-End Promise: Taking Stock of Where We Arecited by 77
  9. 2018Pre-training on high-resource speech recognition improves low-resource speech-to-text translationcited by 70
  10. 2018End-to-End Automatic Speech Translation of Audiobookscited by 65
  11. 2022Unified Speech-Text Pre-training for Speech Translation and Recognitioncited by 64
  12. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
  13. 2018Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translationcited by 57
  14. 2022STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translationcited by 57
  15. 2020Bridging the Modality Gap for Speech-to-Text Translationcited by 55
  16. 2018Tied Multitask Learning for Neural Speech Translationcited by 54
  17. 2021Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference?cited by 54
  18. 2021Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Taskcited by 54
  19. 2020A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Taskscited by 51
  20. 2021Learning Shared Semantic Space for Speech-to-Text Translationcited by 49
  21. 2021Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoderscited by 48
  22. 2021End-to-end Speech Translation via Cross-modal Progressive Trainingcited by 47
  23. 2021CTC-based Compression for Direct Speech Translationcited by 45
  24. 2022SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-trainingcited by 40
  25. 2019Improving Sequence-to-Sequence Learning via Optimal Transportcited by 33
  26. 2020End-to-End Speech-Translation with Knowledge Distillation: FBK@IWSLT2020cited by 31
  27. 2022Sample, Translate, Recombine: Leveraging Audio Alignments for Data Augmentation in End-to-end Speech Translationcited by 27
  28. 2023Pre-training for Speech Translation: CTC Meets Optimal Transportcited by 25
  29. 2020Multilingual Machine Translation: Closing the Gap between Shared and Language-specific Encoder-Decoderscited by 23
  30. 2022WACO: Word-Aligned Contrastive Learning for Speech Translationcited by 21
  31. 2021Using Optimal Transport as Alignment Objective for fine-tuning Multilingual Contextualized Embeddingscited by 19
  32. 2021End-to-End Speech Translation with Pre-trained Models and Adapters: UPC at IWSLT 2021cited by 16
  33. 2022T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translationcited by 16
  34. 2023Understanding and Bridging the Modality Gap for Speech Translationcited by 15
  35. 2020Worse WER, but Better BLEU? Leveraging Word Embedding as Intermediate in Multitask End-to-End Speech Translationcited by 12
  36. 2020Enabling Zero-shot Multilingual Spoken Language Translation with Language-Specific Encoders and Decoderscited by 10
  37. 2023Improving Zero-shot Multilingual Neural Machine Translation by Leveraging Cross-lingual Consistency Regularizationcited by 10
  38. 2023NAVER LABS Europe's Multilingual Speech Translation Systems for the IWSLT 2023 Low-Resource Trackcited by 9

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 8; 2026: 2.

14 papers.

Sort:

By area:

  1. 2024Speech is More Than Words: Do Speech-to-Text Translation Systems Leverage Prosody?cited by 13
  2. 2024How do Multimodal Foundation Models Encode Text and Speech? An Analysis of Cross-Lingual and Cross-Modal Representationscited by 8
  3. 2024Zero-resource Speech Translation and Recognition with LLMscited by 6
  4. 2025Prepending or Cross-Attention for Speech-to-Text? An Empirical Comparisoncited by 4
  5. 2025Scheduled Interleaved Speech-Text Training for Speech-to-Speech Translation with LLMscited by 3
  6. 2025How to Evaluate Speech Translation with Source-Aware Neural MT Metricscited by 3
  7. 2025MCAT: Scaling Many-to-Many Speech-to-Text Translation with MLLMs to 70 Languagescited by 3
  8. 2024Sequential Contrastive Audio-Visual Learningcited by 2
  9. 2025Adaptive Inner Speech-Text Alignment for LLM-based Speech Translationcited by 2
  10. 2025Optimal Transport Regularization for Speech Text Alignment in Spoken Language Modelscited by 2
  11. 2025Improving Language and Modality Transfer in Translation by Character-level Modelingcited by 1
  12. 2025HITSZ's End-To-End Speech Translation Systems Combining Sequence-to-Sequence Auto Speech Recognition Model and Indic Large Language Model for IWSLT 2025 in Indic Trackcited by 0
2 more
  1. 2026Parameter- and Bandwidth-Efficient Edge--cloud Many-to-Many Speech-to-Text Translationcited by 0
  2. 2026Breaking the Curse of Multilinguality in Many-to-Many Speech-to-Text Translation via a Resource-Aware Mixture of Speech Encoderscited by 0