mlsanityData to 7 Oct 2026

An Empirical Study of Consistency Regularization for End-to-End Speech-to-Text Translation

arXiv 2308.14482 ↗First posted 28 Aug 2023cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Consistency regularization methods, such as R-Drop (Liang et al., 2021) and CrossConST (Gao et al., 2023), have achieved impressive supervised and zero-shot performance in the neural machine translation (NMT) field. Can we also boost end-to-end (E2E) speech-to-text translation (ST) by leveraging consistency regularization? In this paper, we conduct empirical studies on intra-modal and cross-modal consistency and propose two training strategies, SimRegCR and SimZeroCR, for E2E ST in regular and zero-shot scenarios. Experiments on the MuST-C benchmark show that our approaches achieve state-of-the-art (SOTA) performance in most translation directions. The analyses prove that regularization brought by the intra-modal consistency, instead of modality gap, is crucial for the regular E2E ST, and the cross-modal consistency could close the modality gap and boost the zero-shot E2E ST performance.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 41 of its 55 references that are in the map. First posted on arXiv 28 Aug 2023.

Builds on

41 of 55 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  4. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  6. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  7. 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
  8. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 174
  9. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
  10. 2018Pre-training on high-resource speech recognition improves low-resource speech-to-text translationcited by 70
  11. 2022Unified Speech-Text Pre-training for Speech Translation and Recognitioncited by 64
  12. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
29 more
  1. 2019End-to-End Speech Translation with Knowledge Distillationcited by 58
  2. 2020Curriculum Pre-training for End-to-End Speech Translationcited by 57
  3. 2022STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translationcited by 57
  4. 2020Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translationcited by 54
  5. 2021Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Taskcited by 54
  6. 2021Lightweight Adapter Tuning for Multilingual Speech Translationcited by 53
  7. 2020A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Taskscited by 51
  8. 2021Learning Shared Semantic Space for Speech-to-Text Translationcited by 49
  9. 2021Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoderscited by 48
  10. 2021End-to-end Speech Translation via Cross-modal Progressive Trainingcited by 47
  11. 2022SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-trainingcited by 40
  12. 2020End-to-End Speech-Translation with Knowledge Distillation: FBK@IWSLT2020cited by 31
  13. 2021Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translationcited by 27
  14. 2022Sample, Translate, Recombine: Leveraging Audio Alignments for Data Augmentation in End-to-end Speech Translationcited by 27
  15. 2023CMOT: Cross-modal Mixup via Optimal Transport for Speech Translationcited by 22
  16. 2022M3ST: Mix at Three Levels for Speech Translationcited by 21
  17. 2022WACO: Word-Aligned Contrastive Learning for Speech Translationcited by 21
  18. 2017Neural Lattice-to-Sequence Models for Uncertain Inputscited by 20
  19. 2019Lattice Transformer for Speech Translationcited by 15
  20. 2021Speechformer: Reducing Information Loss in Direct Speech Translationcited by 15
  21. 2023Understanding and Bridging the Modality Gap for Speech Translationcited by 15
  22. 2019Self-Attentional Models for Lattice Inputscited by 12
  23. 2020Enabling Zero-shot Multilingual Spoken Language Translation with Language-Specific Encoders and Decoderscited by 10
  24. 2023Improving Zero-shot Multilingual Neural Machine Translation by Leveraging Cross-lingual Consistency Regularizationcited by 10
  25. 2023Back Translation for Speech-to-text Translation Without Transcriptscited by 9
  26. 2022Bi-SimCut: A Simple Strategy for Boosting Neural Machine Translationcited by 8
  27. 2023Improving Speech Translation by Cross-Modal Multi-Grained Contrastive Learningcited by 7
  28. 2022Discrete Cross-Modal Alignment Enables Zero-Shot Speech Translationcited by 6
  29. 2023Modality Adaption or Regularization? A Case Study on End-to-End Speech Translationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 1.

6 papers.

  1. 2024Pushing the Limits of Zero-shot End-to-End Speech Translationcited by 14
  2. 2025Speech Translation Refinement using Large Language Modelscited by 2
  3. 2024Transducer Consistency Regularization for Speech to Text Applicationscited by 1
  4. 2025Interpretable Style Takagi-Sugeno-Kang Fuzzy Clusteringcited by 0
  5. 2025Optimal Multi-Task Learning at Regularization Horizon for Speech Translation Taskcited by 0
  6. 2026SimReg: Achieving Higher Performance in the Pretraining via Embedding Similarity Regularizationcited by 0