mlsanityData to 7 Oct 2026

Understanding and Bridging the Modality Gap for Speech Translation

arXiv 2305.08706 ↗First posted 15 May 2023cs.CL cs.SD eess.AScited by 15

Not on the map. Why?

Abstract (arXiv)

How to achieve better end-to-end speech translation (ST) by leveraging (text) machine translation (MT) data? Among various existing techniques, multi-task learning is one of the effective ways to share knowledge between ST and MT in which additional MT data can help to learn source-to-target mapping. However, due to the differences between speech and text, there is always a gap between ST and MT. In this paper, we first aim to understand this modality gap from the target-side representation differences, and link the modality gap to another well-known problem in neural machine translation: exposure bias. We find that the modality gap is relatively small during training except for some difficult cases, but keeps increasing during inference due to the cascading effect. To address these problems, we propose the Cross-modal Regularization with Scheduled Sampling (Cress) method. Specifically, we regularize the output predictions of ST and MT, whose target-side contexts are derived by sampling between ground truth words and self-generated words with a varying probability. Furthermore, we introduce token-level adaptive training which assigns different training weights to target tokens to handle difficult cases with large modality gaps. Experiments and analysis show that our approach effectively bridges the modality gap, and achieves promising results in all eight directions of the MuST-C dataset.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 59 of its 86 references that are in the map. First posted on arXiv 15 May 2023.

Builds on

59 of 86 references.

References · 59

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  6. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  7. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  8. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  9. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  10. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  11. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  12. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
38 more
  1. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  2. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  3. 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
  4. 2016An Actor-Critic Algorithm for Sequence Predictioncited by 158
  5. 2014A* Samplingcited by 150
  6. 2020On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translationcited by 127
  7. 2019Bridging the Gap between Training and Inference for Neural Machine Translationcited by 123
  8. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  9. 2020VoxLingua107: a Dataset for Spoken Language Recognitioncited by 118
  10. 2015Minimum Risk Training for Neural Machine Translationcited by 114
  11. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
  12. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  13. 2022Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generationcited by 106
  14. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  15. 2020Speech Translation and the End-to-End Promise: Taking Stock of Where We Arecited by 77
  16. 2018A Study of Reinforcement Learning for Neural Machine Translationcited by 74
  17. 2020CoVoST 2 and Massively Multilingual Speech-to-Text Translationcited by 73
  18. 2018Pre-training on high-resource speech recognition improves low-resource speech-to-text translationcited by 70
  19. 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
  20. 2022Unified Speech-Text Pre-training for Speech Translation and Recognitioncited by 64
  21. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
  22. 2019End-to-End Speech Translation with Knowledge Distillationcited by 58
  23. 2018Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translationcited by 57
  24. 2020Curriculum Pre-training for End-to-End Speech Translationcited by 57
  25. 2022STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translationcited by 57
  26. 2020Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translationcited by 54
  27. 2021Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference?cited by 54
  28. 2021Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Taskcited by 54
  29. 2021Lightweight Adapter Tuning for Multilingual Speech Translationcited by 53
  30. 2020A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Taskscited by 51
  31. 2021Learning Shared Semantic Space for Speech-to-Text Translationcited by 49
  32. 2021Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoderscited by 48
  33. 2021Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translationcited by 47
  34. 2021End-to-end Speech Translation via Cross-modal Progressive Trainingcited by 47
  35. 2019A Comparative Study on End-to-end Speech to Text Translationcited by 40
  36. 2022SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-trainingcited by 40
  37. 2019Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translationcited by 39
  38. 2021Large-Scale Self- and Semi-Supervised Learning for Speech Translationcited by 36

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 9; 2024: 4; 2025: 2.

15 papers.

Sort:

  1. 2023CMOT: Cross-modal Mixup via Optimal Transport for Speech Translationcited by 22
  2. 2024Pushing the Limits of Zero-shot End-to-End Speech Translationcited by 14
  3. 2025Understanding the Modality Gap: An Empirical Study on the Speech-Text Alignment Mechanism of Large Speech Language Modelscited by 10
  4. 2023Tuning Large language model for End-to-end Speech Translationcited by 9
  5. 2023DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translationcited by 9
  6. 2023Modality Adaption or Regularization? A Case Study on End-to-End Speech Translationcited by 6
  7. 2023An Empirical Study of Consistency Regularization for End-to-End Speech-to-Text Translationcited by 6
  8. 2023Bridging the Gap between Synthetic and Authentic Images for Multimodal Machine Translationcited by 6
  9. 2023Unified Segment-to-Segment Framework for Simultaneous Sequence Generationcited by 6
  10. 2023Rethinking and Improving Multi-task Learning for End-to-end Speech Translationcited by 6
  11. 2024Can We Achieve High-quality Direct Speech-to-Speech Translation without Parallel Speech Data?cited by 4
  12. 2024CoSTA: Code-Switched Speech Translation using Aligned Speech-Text Interleavingcited by 4
3 more
  1. 2025Speech Translation Refinement using Large Language Modelscited by 2
  2. 2024CTC-based Non-autoregressive Textless Speech-to-Speech Translationcited by 1
  3. 2023Bridging the Gaps of Both Modality and Language: Synchronous Bilingual CTC for Speech Translation and Speech Recognitioncited by 0