mlsanityData to 14 Oct 2026

Cross-Modal Multi-Tasking for Speech-to-Text Translation via Hard Parameter Sharing

arXiv 2309.15826 ↗First posted 27 Sep 2023cs.CL cs.SD eess.AScited by 3

Not on the map. Why?

Abstract (arXiv)

Recent works in end-to-end speech-to-text translation (ST) have proposed multi-tasking methods with soft parameter sharing which leverage machine translation (MT) data via secondary encoders that map text inputs to an eventual cross-modal representation. In this work, we instead propose a ST/MT multi-tasking framework with hard parameter sharing in which all model parameters are shared cross-modally. Our method reduces the speech-text modality gap via a pre-processing stage which converts speech and text inputs into two discrete token sequences of similar length -- this allows models to indiscriminately process both modalities simply using a joint vocabulary. With experiments on MuST-C, we demonstrate that our multi-tasking framework improves attentional encoder-decoder, Connectionist Temporal Classification (CTC), transducer, and joint CTC/attention models by an average of +0.5 BLEU without any external MT data. Further, we show that this framework incorporates external MT data, yielding +0.8 BLEU, and also improves transfer learning from pre-trained textual models, yielding +1.8 BLEU.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 44 of its 56 references that are in the map. First posted on arXiv 27 Sep 2023.

Builds on

44 of 56 references.

References · 44

  1. 2017Neural Discrete Representation Learningcited by 4,712
  2. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  4. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  6. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  7. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  8. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 803
  9. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  10. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  11. 2022High Fidelity Neural Audio Compressioncited by 687
  12. 2015Listen, Attend and Spellcited by 543
32 more
  1. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  2. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  3. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  4. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 195
  5. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 180
  6. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 158
  7. 2021Direct speech-to-speech translation with discrete unitscited by 149
  8. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 145
  9. 2023End-to-End Speech Recognition: A Surveycited by 129
  10. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  11. 2023Textually Pretrained Speech Language Modelscited by 101
  12. 2019Neural Machine Translation: A Review and Surveycited by 100
  13. 2022E-Branchformer: Branchformer with Enhanced merging for speech recognitioncited by 92
  14. 2020ESPnet-ST: All-in-One Speech Translation Toolkitcited by 82
  15. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  16. 2022Unified Speech-Text Pre-training for Speech Translation and Recognitioncited by 64
  17. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
  18. 2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
  19. 2022STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translationcited by 57
  20. 2021CTC-based Compression for Direct Speech Translationcited by 45
  21. 2021Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translationcited by 28
  22. 2023Exploration of Efficient End-to-End ASR using Discretized Input from Self-Supervised Learningcited by 28
  23. 2022Revisiting End-to-End Speech-to-Text Translation From Scratchcited by 26
  24. 2022CTC Alignments Improve Autoregressive Translationcited by 25
  25. 2020DiscreTalk: Text-to-Speech as a Machine Translation Problemcited by 21
  26. 2022M3ST: Mix at Three Levels for Speech Translationcited by 21
  27. 2022WACO: Word-Aligned Contrastive Learning for Speech Translationcited by 21
  28. 2022Integrating Text Inputs For Training and Adapting RNN Transducer ASR Modelscited by 20
  29. 2022Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translationcited by 20
  30. 2022Textless Direct Speech-to-Speech Translation with Discrete Speech Representationcited by 19
  31. 2018Multi-Modal Data Augmentation for End-to-End ASRcited by 18
  32. 2023ESPnet-ST-v2: Multipurpose Spoken Language Translation Toolkitcited by 10

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 1.

3 papers.

  1. 2024Representation Purification for End-to-End Speech Translationcited by 3
  2. 2024Compact Speech Translation Models via Discrete Speech Units Pretrainingcited by 1
  3. 2025Optimal Multi-Task Learning at Regularization Horizon for Speech Translation Taskcited by 0