mlsanityData to 7 Oct 2026

CMOT: Cross-modal Mixup via Optimal Transport for Speech Translation

arXiv 2305.14635 ↗First posted 24 May 2023cs.CL cs.AI cs.SDcited by 22

Not on the map. Why?

Abstract (arXiv)

End-to-end speech translation (ST) is the task of translating speech signals in the source language into text in the target language. As a cross-modal task, end-to-end ST is difficult to train with limited data. Existing methods often try to transfer knowledge from machine translation (MT), but their performances are restricted by the modality gap between speech and text. In this paper, we propose Cross-modal Mixup via Optimal Transport CMOT to overcome the modality gap. We find the alignment between speech and text sequences via optimal transport and then mix up the sequences from different modalities at a token level using the alignment. Experiments on the MuST-C ST benchmark demonstrate that CMOT achieves an average BLEU of 30.0 in 8 translation directions, outperforming previous methods. Further analysis shows CMOT can adaptively find the alignment between modalities, which helps alleviate the modality gap between speech and text. Code is publicly available at https://github.com/ictnlp/CMOT.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 49 of its 71 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

49 of 71 references.

References · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  7. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  8. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  9. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  10. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  11. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  12. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 174
37 more
  1. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
  2. 2020ESPnet-ST: All-in-One Speech Translation Toolkitcited by 82
  3. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  4. 2018Pre-training on high-resource speech recognition improves low-resource speech-to-text translationcited by 70
  5. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
  6. 2019End-to-End Speech Translation with Knowledge Distillationcited by 58
  7. 2020Curriculum Pre-training for End-to-End Speech Translationcited by 57
  8. 2022STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translationcited by 57
  9. 2020Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translationcited by 54
  10. 2020A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Taskscited by 51
  11. 2021Learning Shared Semantic Space for Speech-to-Text Translationcited by 49
  12. 2021Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoderscited by 48
  13. 2021Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translationcited by 47
  14. 2021End-to-end Speech Translation via Cross-modal Progressive Trainingcited by 47
  15. 2022SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-trainingcited by 40
  16. 2019Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translationcited by 39
  17. 2020Self-Training for End-to-End Speech Translationcited by 36
  18. 2021Large-Scale Self- and Semi-Supervised Learning for Speech Translationcited by 36
  19. 2020Word Rotator's Distancecited by 32
  20. 2020End-to-End Speech-Translation with Knowledge Distillation: FBK@IWSLT2020cited by 31
  21. 2022Enhancing Cross-lingual Transfer by Manifold Mixupcited by 30
  22. 2022Geodesic Multi-Modal Mixup for Robust Fine-Tuningcited by 29
  23. 2019Analyzing ASR pretraining for low-resource speech-to-text translationcited by 27
  24. 2021Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translationcited by 27
  25. 2022Sample, Translate, Recombine: Leveraging Audio Alignments for Data Augmentation in End-to-end Speech Translationcited by 27
  26. 2022SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learningcited by 26
  27. 2020NeurST: Neural Speech Translation Toolkitcited by 23
  28. 2019Fluent Translations from Disfluent Speech in End-to-End Speech Translationcited by 22
  29. 2015Measuring Word Significance using Distributed Representations of Wordscited by 21
  30. 2022T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translationcited by 16
  31. 2023Understanding and Bridging the Modality Gap for Speech Translationcited by 15
  32. 2018Structured-based Curriculum Learning for End-to-end English-Japanese Speech Translationcited by 11
  33. 2020Jointly Trained Transformers models for Spoken Language Translationcited by 10
  34. 2022Improving End-to-end Speech Translation by Leveraging Auxiliary Speech and Text Datacited by 9
  35. 2023Back Translation for Speech-to-text Translation Without Transcriptscited by 9
  36. 2022Tackling data scarcity in speech translation using zero-shot multilingual machine translation techniquescited by 4
  37. 2022Improving Zero-Shot Multilingual Translation with Universal Representations and Cross-Mappingscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 6; 2025: 8; 2026: 1.

22 papers.

Sort:

  1. 2023Dual-view Curricular Optimal Transport for Cross-lingual Cross-modal Retrievalcited by 9
  2. 2023DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translationcited by 9
  3. 2023An Empirical Study of Consistency Regularization for End-to-End Speech-to-Text Translationcited by 6
  4. 2023Bridging the Gap between Synthetic and Authentic Images for Multimodal Machine Translationcited by 6
  5. 2023Unified Segment-to-Segment Framework for Simultaneous Sequence Generationcited by 6
  6. 2023Rethinking and Improving Multi-task Learning for End-to-end Speech Translationcited by 6
  7. 2024Can We Achieve High-quality Direct Speech-to-Speech Translation without Parallel Speech Data?cited by 4
  8. 2024Temporal Order Preserved Optimal Transport-based Cross-modal Knowledge Transfer Learning for ASRcited by 3
  9. 2024Representation Purification for End-to-End Speech Translationcited by 3
  10. 2025Speech Translation Refinement using Large Language Modelscited by 2
  11. 2025Adaptive Inner Speech-Text Alignment for LLM-based Speech Translationcited by 2
  12. 2025WhisQ: Cross-Modal Representation Learning for Text-to-Music MOS Predictioncited by 2
10 more
  1. 2025Optimal Transport Regularization for Speech Text Alignment in Spoken Language Modelscited by 2
  2. 2025DTW-Align: Bridging the Modality Gap in End-to-End Speech Translation with Dynamic Time Warping Alignmentcited by 2
  3. 2024CTC-based Non-autoregressive Textless Speech-to-Speech Translationcited by 1
  4. 2024PBVS 2024 Solution: Self-Supervised Learning and Sampling Strategies for SAR Classification in Extreme Long-Tail Distributioncited by 1
  5. 2025Cross-modal Knowledge Transfer Learning as Graph Matching Based on Optimal Transport for ASRcited by 1
  6. 2023Bridging the Gaps of Both Modality and Language: Synchronous Bilingual CTC for Speech Translation and Speech Recognitioncited by 0
  7. 2024COTET: Cross-view Optimal Transport for Knowledge Graph Entity Typingcited by 0
  8. 2025Interpretable Style Takagi-Sugeno-Kang Fuzzy Clusteringcited by 0
  9. 2025scMRDR: A scalable and flexible framework for unpaired single-cell multi-omics data integrationcited by 0
  10. 2026Optimal transport meets speech: a tutorial reviewcited by 0