mlsanityData to 7 Oct 2026

M3ST: Mix at Three Levels for Speech Translation

arXiv 2212.03657 ↗First posted 7 Dec 2022cs.CL cs.SD eess.AScited by 21

Not on the map. Why?

Abstract (arXiv)

How to solve the data scarcity problem for end-to-end speech-to-text translation (ST)? It's well known that data augmentation is an efficient method to improve performance for many tasks by enlarging the dataset. In this paper, we propose Mix at three levels for Speech Translation (M^3ST) method to increase the diversity of the augmented training corpus. Specifically, we conduct two phases of fine-tuning based on a pre-trained model using external machine translation (MT) data. In the first stage of fine-tuning, we mix the training corpus at three levels, including word level, sentence level and frame level, and fine-tune the entire model with mixed data. At the second stage of fine-tuning, we take both original speech sequences and original text sequences in parallel into the model to fine-tune the network, and use Jensen-Shannon divergence to regularize their outputs. Experiments on MuST-C speech translation benchmark and analysis show that M^3ST outperforms current strong baselines and achieves state-of-the-art results on eight directions with an average BLEU of 29.9.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 19 of its 26 references that are in the map. First posted on arXiv 7 Dec 2022.

Builds on

19 of 26 references.

References · 19

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  4. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  8. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 174
  9. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
  10. 2022STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translationcited by 57
  11. 2021Learning Shared Semantic Space for Speech-to-Text Translationcited by 49
  12. 2021Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoderscited by 48
7 more
  1. 2021End-to-end Speech Translation via Cross-modal Progressive Trainingcited by 47
  2. 2021MixSpeech: Data Augmentation for Low-resource Automatic Speech Recognitioncited by 38
  3. 2022Sample, Translate, Recombine: Leveraging Audio Alignments for Data Augmentation in End-to-end Speech Translationcited by 27
  4. 2022Revisiting End-to-End Speech-to-Text Translation From Scratchcited by 26
  5. 2022GigaST: A 10,000-hour Pseudo Speech Translation Corpuscited by 20
  6. 2021Regularizing End-to-End Speech Translation with Triangular Decomposition Agreementcited by 16
  7. 2020SkinAugment: Auto-Encoding Speaker Conversions for Automatic Speech Translationcited by 13

Built on by

Cited by year

Citing papers per year: 2023: 19; 2024: 2.

21 papers.

Sort:

By area:

  1. 2023Implicit Neural Representation for Cooperative Low-light Image Enhancementcited by 92
  2. 2023CRoSS: Diffusion Model Makes Controllable, Robust and Secure Image Steganographycited by 50
  3. 2024Enhancing Dialogue State Tracking Models through LLM-backed User-Agents Simulationcited by 24
  4. 2023DUB: Discrete Unit Back-translation for Speech Translationcited by 18
  5. 2023PoseRAC: Pose Saliency Transformer for Repetitive Action Countingcited by 17
  6. 2023A Frustratingly Easy Plug-and-Play Detection-and-Reasoning Module for Chinese Spelling Checkcited by 15
  7. 2023ComSL: A Composite Speech-Language Model for End-to-End Speech-to-Text Translationcited by 9
  8. 2023CTC-based Non-autoregressive Speech Translationcited by 9
  9. 2023Tuning Large language model for End-to-end Speech Translationcited by 9
  10. 2023FiTs: Fine-grained Two-stage Training for Knowledge-aware Question Answeringcited by 8
  11. 2023Out-of-Distributed Semantic Pruning for Robust Semi-Supervised Learningcited by 7
  12. 2023MOSPC: MOS Prediction Based on Pairwise Comparisoncited by 7
9 more
  1. 2023Modality Adaption or Regularization? A Case Study on End-to-End Speech Translationcited by 6
  2. 2023An Empirical Study of Consistency Regularization for End-to-End Speech-to-Text Translationcited by 6
  3. 2023Rethinking and Improving Multi-task Learning for End-to-end Speech Translationcited by 6
  4. 2023ML-LMCL: Mutual Learning and Large-Margin Contrastive Learning for Improving ASR Robustness in Spoken Language Understandingcited by 6
  5. 2023CLEME: Debiasing Multi-reference Evaluation for Grammatical Error Correctioncited by 5
  6. 2023Cross-Modal Multi-Tasking for Speech-to-Text Translation via Hard Parameter Sharingcited by 3
  7. 2023Bridging the Gaps of Both Modality and Language: Synchronous Bilingual CTC for Speech Translation and Speech Recognitioncited by 0
  8. 2023I2KD-SLU: An Intra-Inter Knowledge Distillation Framework for Zero-Shot Cross-Lingual Spoken Language Understandingcited by 0
  9. 2024Revisiting Interpolation Augmentation for Speech-to-Text Generationcited by 0