mlsanityData to 14 Oct 2026

ComSL: A Composite Speech-Language Model for End-to-End Speech-to-Text Translation

arXiv 2305.14838 ↗First posted 24 May 2023cs.CL cs.SD eess.AScited by 9

Not on the map. Why?

Abstract (arXiv)

Joint speech-language training is challenging due to the large demand for training data and GPU consumption, as well as the modality gap between speech and language. We present ComSL, a speech-language model built atop a composite architecture of public pretrained speech-only and language-only models and optimized data-efficiently for spoken language tasks. Particularly, we propose to incorporate cross-modality learning into transfer learning and conduct them simultaneously for downstream tasks in a multi-task learning manner. Our approach has demonstrated effectiveness in end-to-end speech-to-text translation tasks, achieving a new state-of-the-art average BLEU score of 31.5 on the multilingual speech to English text translation task for 21 languages, as measured on the public CoVoST2 evaluation set.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 41 of its 47 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

41 of 47 references.

References · 41

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  4. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  5. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  6. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  7. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  8. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  9. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 231
  10. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 180
  11. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  12. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
29 more
  1. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  2. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  3. 2020Speech Translation and the End-to-End Promise: Taking Stock of Where We Arecited by 77
  4. 2018Pre-training on high-resource speech recognition improves low-resource speech-to-text translationcited by 70
  5. 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
  6. 2022Unified Speech-Text Pre-training for Speech Translation and Recognitioncited by 64
  7. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
  8. 2019End-to-End Speech Translation with Knowledge Distillationcited by 58
  9. 2018Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translationcited by 57
  10. 2020Curriculum Pre-training for End-to-End Speech Translationcited by 57
  11. 2022STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translationcited by 57
  12. 2021Lightweight Adapter Tuning for Multilingual Speech Translationcited by 54
  13. 2021Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Taskcited by 54
  14. 2021Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoderscited by 48
  15. 2021Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translationcited by 47
  16. 2021End-to-end Speech Translation via Cross-modal Progressive Trainingcited by 47
  17. 2019A Comparative Study on End-to-end Speech to Text Translationcited by 40
  18. 2022SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-trainingcited by 40
  19. 2019Bridging the Gap between Pre-Training and Fine-Tuning for End-to-End Speech Translationcited by 39
  20. 2021Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translationcited by 28
  21. 2019Analyzing ASR pretraining for low-resource speech-to-text translationcited by 27
  22. 2022Sample, Translate, Recombine: Leveraging Audio Alignments for Data Augmentation in End-to-end Speech Translationcited by 27
  23. 2019Fluent Translations from Disfluent Speech in End-to-End Speech Translationcited by 22
  24. 2022M3ST: Mix at Three Levels for Speech Translationcited by 21
  25. 2022WACO: Word-Aligned Contrastive Learning for Speech Translationcited by 21
  26. 2019On Using SpecAugment for End-to-End Speech Translationcited by 19
  27. 2022A Weakly-Supervised Streaming Multilingual Speech Model with Truly Zero-Shot Capabilitycited by 12
  28. 2022Mu2SLAM: Multitask, Multilingual Speech and Language Modelscited by 7
  29. 2021FST: the FAIR Speech Translation System for the IWSLT21 Multilingual Shared Taskcited by 6

Built on by

Cited by year

Citing papers per year: 2024: 7; 2025: 2.

9 papers.

  1. 2024GenTranslate: Large Language Models are Generative Multilingual Speech and Machine Translatorscited by 22
  2. 2024TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservationcited by 10
  3. 2024Investigating Decoder-only Large Language Models for Speech-to-text Translationcited by 9
  4. 2024Recent Advances in End-to-End Simultaneous Speech Translationcited by 5
  5. 2024Enhancing Dialogue Speech Recognition with Robust Contextual Awareness via Noise Representation Learningcited by 2
  6. 2024Transcribing and Translating, Fast and Slow: Joint Speech Translation and Recognitioncited by 2
  7. 2024A Mel Spectrogram Enhancement Paradigm Based on CWT in Speech Synthesiscited by 1
  8. 2025Novel Parasitic Dual-Scale Modeling for Efficient and Accurate Multilingual Speech Translationcited by 0
  9. 2025Whisper-UT: A Unified Translation Framework for Speech and Textcited by 0