mlsanityData to 14 Oct 2026

Optimal Multi-Task Learning at Regularization Horizon for Speech Translation Task

arXiv 2509.09701 ↗First posted 4 Sep 2025cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

End-to-end speech-to-text translation typically suffers from the scarcity of paired speech-text data. One way to overcome this shortcoming is to utilize the bitext data from the Machine Translation (MT) task and perform Multi-Task Learning (MTL). In this paper, we formulate MTL from a regularization perspective and explore how sequences can be regularized within and across modalities. By thoroughly investigating the effect of consistency regularization (different modality) and R-drop (same modality), we show how they respectively contribute to the total regularization. We also demonstrate that the coefficient of MT loss serves as another source of regularization in the MTL setting. With these three sources of regularization, we introduce the optimal regularization contour in the high-dimensional space, called the regularization horizon. Experiments show that tuning the hyperparameters within the regularization horizon achieves near state-of-the-art performance on the MuST-C dataset.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 28 of its 36 references that are in the map. First posted on arXiv 4 Sep 2025.

Builds on

28 of 36 references.

References · 28

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  4. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  5. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  6. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,060
  7. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  8. 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
  9. 2019Consistency Regularization for Generative Adversarial Networkscited by 136
  10. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
  11. 2020A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generationcited by 72
  12. 2018End-to-End Automatic Speech Translation of Audiobookscited by 65
16 more
  1. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
  2. 2022STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translationcited by 57
  3. 2018Tied Multitask Learning for Neural Speech Translationcited by 54
  4. 2021Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference?cited by 54
  5. 2021Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Taskcited by 54
  6. 2021Learning Shared Semantic Space for Speech-to-Text Translationcited by 49
  7. 2021Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoderscited by 48
  8. 2021End-to-end Speech Translation via Cross-modal Progressive Trainingcited by 47
  9. 2019Improving Zero-shot Translation with Language-Independent Constraintscited by 33
  10. 2020Self-Supervised Representations Improve End-to-End Speech Translationcited by 21
  11. 2022WACO: Word-Aligned Contrastive Learning for Speech Translationcited by 21
  12. 2023Improving Speech Translation by Cross-Modal Multi-Grained Contrastive Learningcited by 7
  13. 2023Modality Adaption or Regularization? A Case Study on End-to-End Speech Translationcited by 6
  14. 2023An Empirical Study of Consistency Regularization for End-to-End Speech-to-Text Translationcited by 6
  15. 2023Cross-Modal Multi-Tasking for Speech-to-Text Translation via Hard Parameter Sharingcited by 3
  16. 2022Efficient Speech Translation with Dynamic Latent Perceiverscited by 2

Built on by

Not cited yet.