mlsanityData to 14 Oct 2026

Rethinking and Improving Multi-task Learning for End-to-end Speech Translation

arXiv 2311.03810 ↗First posted 7 Nov 2023cs.CL cs.AI cs.SDcited by 6

Not on the map. Why?

Abstract (arXiv)

Significant improvements in end-to-end speech translation (ST) have been achieved through the application of multi-task learning. However, the extent to which auxiliary tasks are highly consistent with the ST task, and how much this approach truly helps, have not been thoroughly studied. In this paper, we investigate the consistency between different tasks, considering different times and modules. We find that the textual encoder primarily facilitates cross-modal conversion, but the presence of noise in speech impedes the consistency between text and speech representations. Furthermore, we propose an improved multi-task learning (IMTL) approach for the ST task, which bridges the modal gap by mitigating the difference in length and representation. We conduct experiments on the MuST-C dataset. The results demonstrate that our method attains state-of-the-art results. Moreover, when additional data is used, we achieve the new SOTA result on MuST-C English to Spanish task with 20.8% of the training time required by the current SOTA method.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 35 of its 52 references that are in the map. First posted on arXiv 7 Nov 2023.

Builds on

35 of 52 references.

References · 35

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  3. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,332
  4. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  6. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  7. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 834
  8. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
  9. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 176
  10. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  11. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
  12. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
23 more
  1. 2020Curriculum Pre-training for End-to-End Speech Translationcited by 57
  2. 2022STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translationcited by 57
  3. 2020Bridging the Modality Gap for Speech-to-Text Translationcited by 55
  4. 2018Tied Multitask Learning for Neural Speech Translationcited by 54
  5. 2020Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translationcited by 54
  6. 2020A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Taskscited by 51
  7. 2021Learning Shared Semantic Space for Speech-to-Text Translationcited by 49
  8. 2021Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoderscited by 48
  9. 2021Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translationcited by 47
  10. 2021End-to-end Speech Translation via Cross-modal Progressive Trainingcited by 47
  11. 2019A Comparative Study on End-to-end Speech to Text Translationcited by 40
  12. 2022SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-trainingcited by 40
  13. 2019Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translationcited by 39
  14. 2020Consecutive Decoding for Speech-to-text Translationcited by 28
  15. 2022Revisiting End-to-End Speech-to-Text Translation From Scratchcited by 26
  16. 2023Pre-training for Speech Translation: CTC Meets Optimal Transportcited by 25
  17. 2023CMOT: Cross-modal Mixup via Optimal Transport for Speech Translationcited by 22
  18. 2022M3ST: Mix at Three Levels for Speech Translationcited by 21
  19. 2023Understanding and Bridging the Modality Gap for Speech Translationcited by 15
  20. 2023Recent Advances in Direct Speech-to-text Translationcited by 15
  21. 2020Jointly Trained Transformers models for Spoken Language Translationcited by 10
  22. 2022Improving End-to-end Speech Translation by Leveraging Auxiliary Speech and Text Datacited by 9
  23. 2023Bridging the Granularity Gap for Acoustic Modelingcited by 2

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1; 2025: 4.

6 papers.

  1. 2025Soundwave: Less is More for Speech-Text Alignment in LLMscited by 7
  2. 2023Soft Alignment of Modality Space for End-to-end Speech Translationcited by 4
  3. 2024CoSTA: Code-Switched Speech Translation using Aligned Speech-Text Interleavingcited by 4
  4. 2025Improving Language and Modality Transfer in Translation by Character-level Modelingcited by 2
  5. 2025Leveraging Unit Language Guidance to Advance Speech Modeling in Textless Speech-to-Speech Translationcited by 1
  6. 2025Whisper-UT: A Unified Translation Framework for Speech and Textcited by 0