mlsanityData to 7 Oct 2026

STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation

arXiv 2203.10426 ↗First posted 20 Mar 2022cs.CL cs.AIcited by 57

Not on the map. Why?

Abstract (arXiv)

How to learn a better speech representation for end-to-end speech-to-text translation (ST) with limited labeled data? Existing techniques often attempt to transfer powerful machine translation (MT) capabilities to ST, but neglect the representation discrepancy across modalities. In this paper, we propose the Speech-TExt Manifold Mixup (STEMM) method to calibrate such discrepancy. Specifically, we mix up the representation sequences of different modalities, and take both unimodal speech sequences and multimodal mixed sequences as input to the translation model in parallel, and regularize their output predictions with a self-learning framework. Experiments on MuST-C speech translation benchmark and further analysis show that our method effectively alleviates the cross-modal representation discrepancy, and achieves significant improvements over a strong baseline on eight translation directions.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). Builds on 56 of its 76 references that are in the map. First posted on arXiv 20 Mar 2022.

Builds on

56 of 76 references.

References · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  6. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  7. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  8. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  9. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  10. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  11. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  12. 2020MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classificationcited by 186
38 more
  1. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 174
  2. 2019Bridging the Gap between Training and Inference for Neural Machine Translationcited by 123
  3. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  4. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
  5. 2020ESPnet-ST: All-in-One Speech Translation Toolkitcited by 82
  6. 2018Pre-training on high-resource speech recognition improves low-resource speech-to-text translationcited by 70
  7. 2018End-to-End Automatic Speech Translation of Audiobookscited by 65
  8. 2020Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Informationcited by 64
  9. 2019End-to-End Speech Translation with Knowledge Distillationcited by 58
  10. 2018Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translationcited by 57
  11. 2020Curriculum Pre-training for End-to-End Speech Translationcited by 57
  12. 2020Bridging the Modality Gap for Speech-to-Text Translationcited by 55
  13. 2020Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translationcited by 54
  14. 2021Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Taskcited by 54
  15. 2020A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Taskscited by 51
  16. 2021Learning Shared Semantic Space for Speech-to-Text Translationcited by 49
  17. 2021Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoderscited by 48
  18. 2021Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translationcited by 47
  19. 2021End-to-end Speech Translation via Cross-modal Progressive Trainingcited by 47
  20. 2019A Comparative Study on End-to-end Speech to Text Translationcited by 40
  21. 2019Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translationcited by 39
  22. 2021MixSpeech: Data Augmentation for Low-resource Automatic Speech Recognitioncited by 38
  23. 2020Self-Training for End-to-End Speech Translationcited by 36
  24. 2018Towards Robust Neural Machine Translationcited by 35
  25. 2020End-to-End Speech-Translation with Knowledge Distillation: FBK@IWSLT2020cited by 31
  26. 2020Consecutive Decoding for Speech-to-text Translationcited by 28
  27. 2019Analyzing ASR pretraining for low-resource speech-to-text translationcited by 27
  28. 2021Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translationcited by 27
  29. 2022Neural Machine Translation with Phrase-Level Universal Visual Representationscited by 26
  30. 2020Adaptive Feature Selection for End-to-End Speech Translationcited by 25
  31. 2019Fluent Translations from Disfluent Speech in End-to-End Speech Translationcited by 22
  32. 2017Neural Lattice-to-Sequence Models for Uncertain Inputscited by 20
  33. 2019On Using SpecAugment for End-to-End Speech Translationcited by 19
  34. 2019Lattice Transformer for Speech Translationcited by 15
  35. 2020Mixup-Transformer: Dynamic Data Augmentation for NLP Taskscited by 15
  36. 2019Self-Attentional Models for Lattice Inputscited by 12
  37. 2018Structured-based Curriculum Learning for End-to-end English-Japanese Speech Translationcited by 11
  38. 2019Data Efficient Direct Speech-to-Text Translation with Modality Agnostic Meta-Learningcited by 11

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 11; 2023: 28; 2024: 8; 2025: 8; 2026: 2.

57 papers.

Sort:

  1. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
  2. 2022SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-trainingcited by 40
  3. 2023AlignAtt: Using Attention-based Audio-Translation Alignments as a Guide for Simultaneous Speech Translationcited by 35
  4. 2022A Survey of Mix-based Data Augmentation: Taxonomy, Methods, Applications, and Explainabilitycited by 30
  5. 2023Pre-training for Speech Translation: CTC Meets Optimal Transportcited by 25
  6. 2023CMOT: Cross-modal Mixup via Optimal Transport for Speech Translationcited by 22
  7. 2022M3ST: Mix at Three Levels for Speech Translationcited by 21
  8. 2022WACO: Word-Aligned Contrastive Learning for Speech Translationcited by 21
  9. 2023DUB: Discrete Unit Back-translation for Speech Translationcited by 18
  10. 2024A Survey on Mixup Augmentations and Beyondcited by 18
  11. 2023Cross-modality Data Augmentation for End-to-End Sign Language Translationcited by 16
  12. 2023Understanding and Bridging the Modality Gap for Speech Translationcited by 15
38 more
  1. 2023Recent Advances in Direct Speech-to-text Translationcited by 15
  2. 2023MixSpeech: Cross-Modality Self-Learning with Audio-Visual Stream Mixup for Visual Speech Translation and Recognitioncited by 14
  3. 2024Pushing the Limits of Zero-shot End-to-End Speech Translationcited by 14
  4. 2023Back Translation for Speech-to-text Translation Without Transcriptscited by 9
  5. 2023ComSL: A Composite Speech-Language Model for End-to-End Speech-to-Text Translationcited by 9
  6. 2023CTC-based Non-autoregressive Speech Translationcited by 9
  7. 2023Tuning Large language model for End-to-end Speech Translationcited by 9
  8. 2023DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translationcited by 9
  9. 2023End-to-End Speech-to-Text Translation: A Surveycited by 9
  10. 2024Contrastive Learning for Task-Independent SpeechLLM-Pretrainingcited by 8
  11. 2023Improving Speech Translation by Cross-Modal Multi-Grained Contrastive Learningcited by 7
  12. 2022Discrete Cross-Modal Alignment Enables Zero-Shot Speech Translationcited by 6
  13. 2022SegAugment: Maximizing the Utility of Speech Translation Data with Segmentation-based Augmentationscited by 6
  14. 2023Adapting Offline Speech Translation Models for Streaming with Future-Aware Distillation and Inferencecited by 6
  15. 2023Modality Adaption or Regularization? A Case Study on End-to-End Speech Translationcited by 6
  16. 2023An Empirical Study of Consistency Regularization for End-to-End Speech-to-Text Translationcited by 6
  17. 2023Bridging the Gap between Synthetic and Authentic Images for Multimodal Machine Translationcited by 6
  18. 2023Unified Segment-to-Segment Framework for Simultaneous Sequence Generationcited by 6
  19. 2023Rethinking and Improving Multi-task Learning for End-to-end Speech Translationcited by 6
  20. 2022Low-resource Neural Machine Translation with Cross-modal Alignmentcited by 5
  21. 2022ULN: Towards Underspecified Vision-and-Language Navigationcited by 5
  22. 2022Beyond Instance Discrimination: Relation-aware Contrastive Self-supervised Learningcited by 4
  23. 2023Soft Alignment of Modality Space for End-to-end Speech Translationcited by 4
  24. 2024Can We Achieve High-quality Direct Speech-to-Speech Translation without Parallel Speech Data?cited by 4
  25. 2024CoSTA: Code-Switched Speech Translation using Aligned Speech-Text Interleavingcited by 4
  26. 2025Prepending or Cross-Attention for Speech-to-Text? An Empirical Comparisoncited by 4
  27. 2022AdaTranS: Adapting with Boundary-based Shrinking for End-to-End Speech Translationcited by 3
  28. 2023Decouple Non-parametric Knowledge Distillation For End-to-end Speech Translationcited by 3
  29. 2023Improving speech translation by fusing speech and textcited by 3
  30. 2023Cross-Modal Multi-Tasking for Speech-to-Text Translation via Hard Parameter Sharingcited by 3
  31. 2023Towards a Deep Understanding of Multilingual End-to-End Speech Translationcited by 3
  32. 2024Representation Purification for End-to-End Speech Translationcited by 3
  33. 2023MixRep: Hidden Representation Mixup for Low-Resource Speech Recognitioncited by 2
  34. 2025Speech Translation Refinement using Large Language Modelscited by 2
  35. 2025Adaptive Inner Speech-Text Alignment for LLM-based Speech Translationcited by 2
  36. 2025DTW-Align: Bridging the Modality Gap in End-to-End Speech Translation with Dynamic Time Warping Alignmentcited by 2
  37. 2024R-BI: Regularized Batched Inputs enhance Incremental Decoding Framework for Low-Latency Simultaneous Speech Translationcited by 1
  38. 2024CTC-based Non-autoregressive Textless Speech-to-Speech Translationcited by 1

50 most cited of 57 All 57 (JSON)