mlsanityData to 14 Oct 2026

Modality Adaption or Regularization? A Case Study on End-to-End Speech Translation

arXiv 2306.07650 ↗First posted 13 Jun 2023cs.CL cs.SD eess.AScited by 6

Not on the map. Why?

Abstract (arXiv)

Pre-training and fine-tuning is a paradigm for alleviating the data scarcity problem in end-to-end speech translation (E2E ST). The commonplace "modality gap" between speech and text data often leads to inconsistent inputs between pre-training and fine-tuning. However, we observe that this gap occurs in the early stages of fine-tuning, but does not have a major impact on the final performance. On the other hand, we find that there has another gap, which we call the "capacity gap": high resource tasks (such as ASR and MT) always require a large model to fit, when the model is reused for a low resource task (E2E ST), it will get a sub-optimal performance due to the over-fitting. In a case study, we find that the regularization plays a more important role than the well-designed modality adaption method, which achieves 29.0 for en-de and 40.3 for en-fr on the MuST-C dataset. Code and models are available at https://github.com/hannlp/TAB.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 20 of its 24 references that are in the map. First posted on arXiv 13 Jun 2023.

Builds on

20 of 24 references.

References · 20

  1. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  2. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  3. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  4. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  5. 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
  6. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  7. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
  8. 2022STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translationcited by 57
  9. 2020Bridging the Modality Gap for Speech-to-Text Translationcited by 55
  10. 2021Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Taskcited by 54
  11. 2021Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoderscited by 48
  12. 2021End-to-end Speech Translation via Cross-modal Progressive Trainingcited by 47
8 more
  1. 2022SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-trainingcited by 40
  2. 2019Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translationcited by 39
  3. 2022M3ST: Mix at Three Levels for Speech Translationcited by 21
  4. 2022WACO: Word-Aligned Contrastive Learning for Speech Translationcited by 21
  5. 2023Understanding and Bridging the Modality Gap for Speech Translationcited by 15
  6. 2022Improving End-to-end Speech Translation by Leveraging Auxiliary Speech and Text Datacited by 9
  7. 2022Bi-SimCut: A Simple Strategy for Boosting Neural Machine Translationcited by 8
  8. 2022AdaTranS: Adapting with Boundary-based Shrinking for End-to-End Speech Translationcited by 3

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 0; 2025: 5.

6 papers.

  1. 2025Understanding the Modality Gap: An Empirical Study on the Speech-Text Alignment Mechanism of Large Speech Language Modelscited by 13
  2. 2023An Empirical Study of Consistency Regularization for End-to-End Speech-to-Text Translationcited by 6
  3. 2025Speech Translation Refinement using Large Language Modelscited by 2
  4. 2025Optimizing Speech Multi-View Feature Fusion through Conditional Computationcited by 0
  5. 2025Interpretable Style Takagi-Sugeno-Kang Fuzzy Clusteringcited by 0
  6. 2025Optimal Multi-Task Learning at Regularization Horizon for Speech Translation Taskcited by 0