# An Empirical Study of Consistency Regularization for End-to-End Speech-to-Text Translation

arXiv: https://arxiv.org/abs/2308.14482 · HTML: https://mlsanity.com/paper/2308.14482/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 55 references that are in the map.
- First posted on arXiv 28 Aug 2023.

## Builds on (most cited first, 30 of 41)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://mlsanity.com/paper/2004.11867/) (2020)
- [R-Drop: Regularized Dropout for Neural Networks](https://mlsanity.com/paper/2106.14448/) (2021)
- [fairseq S2T: Fast Speech-to-Text Modeling with fairseq](https://mlsanity.com/paper/2010.05171/) (2020)
- [Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation](https://mlsanity.com/paper/1612.01744/) (2016)
- [Pre-training on high-resource speech recognition improves low-resource speech-to-text translation](https://mlsanity.com/paper/1809.01431/) (2018)
- [Unified Speech-Text Pre-training for Speech Translation and Recognition](https://mlsanity.com/paper/2204.05409/) (2022)
- [Cross-modal Contrastive Learning for Speech Translation](https://mlsanity.com/paper/2205.02444/) (2022)
- [End-to-End Speech Translation with Knowledge Distillation](https://mlsanity.com/paper/1904.08075/) (2019)
- [Curriculum Pre-training for End-to-End Speech Translation](https://mlsanity.com/paper/2004.10093/) (2020)
- [STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation](https://mlsanity.com/paper/2203.10426/) (2022)
- [Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translation](https://mlsanity.com/paper/2011.00747/) (2020)
- [Lightweight Adapter Tuning for Multilingual Speech Translation](https://mlsanity.com/paper/2106.01463/) (2021)
- [Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task](https://mlsanity.com/paper/2107.05782/) (2021)
- [A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Tasks](https://mlsanity.com/paper/2010.11338/) (2020)
- [Learning Shared Semantic Space for Speech-to-Text Translation](https://mlsanity.com/paper/2105.03095/) (2021)
- [Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoders](https://mlsanity.com/paper/2105.05752/) (2021)
- [End-to-end Speech Translation via Cross-modal Progressive Training](https://mlsanity.com/paper/2104.10380/) (2021)
- [SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training](https://mlsanity.com/paper/2210.03730/) (2022)
- [End-to-End Speech-Translation with Knowledge Distillation: FBK@IWSLT2020](https://mlsanity.com/paper/2006.02965/) (2020)
- [Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translation](https://mlsanity.com/paper/2104.06457/) (2021)
- [Sample, Translate, Recombine: Leveraging Audio Alignments for Data Augmentation in End-to-end Speech Translation](https://mlsanity.com/paper/2203.08757/) (2022)
- [CMOT: Cross-modal Mixup via Optimal Transport for Speech Translation](https://mlsanity.com/paper/2305.14635/) (2023)
- [M3ST: Mix at Three Levels for Speech Translation](https://mlsanity.com/paper/2212.03657/) (2022)
- [WACO: Word-Aligned Contrastive Learning for Speech Translation](https://mlsanity.com/paper/2212.09359/) (2022)
- [Neural Lattice-to-Sequence Models for Uncertain Inputs](https://mlsanity.com/paper/1704.00559/) (2017)

## Built on by (most cited first, 6 of 6)

- [Pushing the Limits of Zero-shot End-to-End Speech Translation](https://mlsanity.com/paper/2402.10422/) (2024)
- [Speech Translation Refinement using Large Language Models](https://mlsanity.com/paper/2501.15090/) (2025)
- [Transducer Consistency Regularization for Speech to Text Applications](https://mlsanity.com/paper/2410.07491/) (2024)
- [Interpretable Style Takagi-Sugeno-Kang Fuzzy Clustering](https://mlsanity.com/paper/2504.05125/) (2025)
- [Optimal Multi-Task Learning at Regularization Horizon for Speech Translation Task](https://mlsanity.com/paper/2509.09701/) (2025)
- [SimReg: Achieving Higher Performance in the Pretraining via Embedding Similarity Regularization](https://mlsanity.com/paper/2605.08809/) (2026)

## Abstract (arXiv)

Consistency regularization methods, such as R-Drop (Liang et al., 2021) and CrossConST (Gao et al., 2023), have achieved impressive supervised and zero-shot performance in the neural machine translation (NMT) field. Can we also boost end-to-end (E2E) speech-to-text translation (ST) by leveraging consistency regularization? In this paper, we conduct empirical studies on intra-modal and cross-modal consistency and propose two training strategies, SimRegCR and SimZeroCR, for E2E ST in regular and zero-shot scenarios. Experiments on the MuST-C benchmark show that our approaches achieve state-of-the-art (SOTA) performance in most translation directions. The analyses prove that regularization brought by the intra-modal consistency, instead of modality gap, is crucial for the regular E2E ST, and the cross-modal consistency could close the modality gap and boost the zero-shot E2E ST performance.