mlsanityData to 7 Oct 2026

Investigating Decoder-only Large Language Models for Speech-to-text Translation

arXiv 2407.03169 ↗First posted 3 Jul 2024cs.CL cs.SD eess.AScited by 9

On the map: LLM reasoning → Audio-language model reasoning

Abstract (arXiv)

Large language models (LLMs), known for their exceptional reasoning capabilities, generalizability, and fluency across diverse domains, present a promising avenue for enhancing speech-related tasks. In this paper, we focus on integrating decoder-only LLMs to the task of speech-to-text translation (S2TT). We propose a decoder-only architecture that enables the LLM to directly consume the encoded speech representation and generate the text translation. Additionally, we investigate the effects of different parameter-efficient fine-tuning techniques and task formulation. Our model achieves state-of-the-art performance on CoVoST 2 and FLEURS among models trained without proprietary data. We also conduct analyses to validate the design choices of our proposed model and bring insights to the integration of LLMs to S2TT.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning. Builds on 27 of its 33 references that are in the map. First posted on arXiv 3 Jul 2024.

Builds on

27 of 33 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  7. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  10. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  11. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  12. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
15 more
  1. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  2. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  3. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  4. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  5. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  6. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  7. 2023Listen, Think, and Understandcited by 206
  8. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 144
  9. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 139
  10. 2023SeamlessM4T: Massively Multilingual & Multimodal Machine Translationcited by 124
  11. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 79
  12. 2023SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translationcited by 75
  13. 2023SLM: Bridge the thin gap between speech and text foundation modelscited by 67
  14. 2019Multilingual End-to-End Speech Translationcited by 37
  15. 2023ComSL: A Composite Speech-Language Model for End-to-End Speech-to-Text Translationcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 2.

9 papers.

  1. 2024Language Model Can Listen While Speakingcited by 62
  2. 2024Task Arithmetic for Language Expansion in Speech Translationcited by 6
  3. 2025Prepending or Cross-Attention for Speech-to-Text? An Empirical Comparisoncited by 4
  4. 2025Speech-to-Text Translation with Phoneme-Augmented CoT: Enhancing Cross-Lingual Transfer in Low-Resource Scenarioscited by 3
  5. 2025Scheduled Interleaved Speech-Text Training for Speech-to-Speech Translation with LLMscited by 3
  6. 2026DOA: Training-Free Decoder-Only Attention Policy for Long-Form Simultaneous Translation with SpeechLLMscited by 1
  7. 2025Transformer-Encoder Trees for Efficient Multilingual Machine Translation and Speech Translationcited by 0
  8. 2025PART: Progressive Alignment Representation Training for Multilingual Speech-To-Text with LLMscited by 0
  9. 2026FBK's Long-form SpeechLLMs for IWSLT 2026 Instruction Followingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond Decoder-only: Large Language Models Can be Good Encoders for Machine Translation
  2. 2023Speech Translation with Large Language Models: An Industrial Practice
  3. 2024Making LLMs Better Many-to-Many Speech-to-Text Translators with Curriculum Learning
  4. 2024Think Big, Generate Quick: LLM-to-SLM for Fast Autoregressive Decoding
  5. 2024A Survey on Efficient Inference for Large Language Models
  6. 2026PROST-LLM: Progressively Enhancing the Speech-to-Speech Translation Capability in LLMs
  7. 2025Bridging the Linguistic Divide: A Survey on Leveraging Large Language Models for Machine Translation
  8. 2024GenTranslate: Large Language Models are Generative Multilingual Speech and Machine Translators