mlsanityData to 7 Oct 2026

Prompting Large Language Models with Speech Recognition Abilities

arXiv 2307.11795 ↗First posted 21 Jul 2023eess.AS cs.AI cs.CLcited by 139

Not on the map. Why?

Abstract (arXiv)

Large language models have proven themselves highly flexible, able to solve a wide range of generative tasks, such as abstractive summarization and open-ended question answering. In this paper we extend the capabilities of LLMs by directly attaching a small audio encoder allowing it to perform speech recognition. By directly prepending a sequence of audial embeddings to the text token embeddings, the LLM can be converted to an automatic speech recognition (ASR) system, and be used in the exact same manner as its textual counterpart. Experiments on Multilingual LibriSpeech (MLS) show that incorporating a conformer encoder into the open sourced LLaMA-7B allows it to outperform monolingual baselines by 18% and perform multilingual speech recognition despite LLaMA being trained overwhelmingly on English text. Furthermore, we perform ablation studies to investigate whether the LLM can be completely frozen during training to maintain its original capabilities, scaling up the audio encoder, and increasing the audio encoder striding to generate fewer embeddings. The results from these studies show that multilingual ASR is possible even when the LLM is frozen or when strides of almost 1 second are used in the audio encoder opening up the possibility for LLMs to operate on long-form audio.

Key facts. Cited by 139 papers in the map (data through 7 Oct 2026). Builds on 23 of its 25 references that are in the map. First posted on arXiv 21 Jul 2023.

Builds on

23 of 25 references.

References · 23

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  10. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  11. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
11 more
  1. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  2. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  3. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  4. 2017Learning multiple visual domains with residual adapterscited by 553
  5. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  6. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  7. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  8. 2023Listen, Think, and Understandcited by 206
  9. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 158
  10. 2021Joint Unsupervised and Supervised Training for Multilingual ASRcited by 39
  11. 2022Improving Fast-slow Encoder based Transducer with Streaming Deliberationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 16; 2024: 65; 2025: 35; 2026: 23.

139 papers.

Sort:

By area:

  1. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  2. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 169
  3. 2024An Embarrassingly Simple Approach for LLM with Strong ASR Capacitycited by 86
  4. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 79
  5. 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 79
  6. 2023SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translationcited by 75
  7. 2025Audio-CoT: Exploring Chain-of-Thought Reasoning in Large Audio Language Modelcited by 60
  8. 2023AudioChatLlama: Towards General-Purpose Speech Abilities for LLMscited by 59
  9. 2024Large Language Models are Strong Audio-Visual Speech Recognition Learnerscited by 40
  10. 2023Can Whisper perform speech-based in-context learning?cited by 35
  11. 2024Distilling an End-to-End Voice Assistant Without Instruction Training Datacited by 35
  12. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 35
38 more
  1. 2024TransLLaMa: LLM-based Simultaneous Translation Systemcited by 34
  2. 2025TimeCAP: Learning to Contextualize, Augment, and Predict Time Series Events with Large Language Model Agentscited by 33
  3. 2025SALM-Duplex: Efficient and Direct Duplex Modeling for Speech-to-Speech Language Modelcited by 32
  4. 2024DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factorscited by 31
  5. 2025Distraction is All You Need for Multimodal Large Language Model Jailbreakingcited by 31
  6. 2023COSMIC: Data Efficient Instruction-tuning For Speech In-Context Learningcited by 30
  7. 2024Large Language Models are Efficient Learners of Noise-Robust Speech Recognitioncited by 30
  8. 2024Exploring Autonomous Agents through the Lens of Large Language Models: A Reviewcited by 30
  9. 2024Where Visual Speech Meets Language: VSP-LLM Framework for Efficient and Context-Aware Visual Speech Processingcited by 29
  10. 2023Speech Translation with Large Language Models: An Industrial Practicecited by 28
  11. 2025Audio Large Language Models Can Be Descriptive Speech Quality Evaluatorscited by 25
  12. 2024Towards Open Respiratory Acoustic Foundation Models: Pretraining and Benchmarkingcited by 23
  13. 2023End-to-End Speech Recognition Contextualization with Large Language Modelscited by 22
  14. 2024It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognitioncited by 22
  15. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 22
  16. 2023Boosting Large Language Model for Speech Synthesis: An Empirical Studycited by 21
  17. 2025MMS-LLaMA: Efficient LLM-based Audio-Visual Speech Recognition with Minimal Multimodal Speech Tokenscited by 19
  18. 2024BESTOW: Efficient and Streamable Speech Language Model with the Best of Two Worlds in GPT and T5cited by 18
  19. 2024Frozen Large Language Models Can Perceive Paralinguistic Aspects of Speechcited by 18
  20. 2024The Sound of Healthcare: Improving Medical Transcription ASR Accuracy with Large Language Modelscited by 17
  21. 2024Chain-of-Scrutiny: Detecting Backdoor Attacks for Large Language Modelscited by 17
  22. 2024Chain-of-Thought Prompting for Speech Translationcited by 17
  23. 2024Performance evaluation of SLAM-ASR: The Good, the Bad, the Ugly, and the Way Forwardcited by 17
  24. 2025ALLM4ADD: Unlocking the Capabilities of Audio Large Language Models for Audio Deepfake Detectioncited by 17
  25. 2024SSR: Alignment-Aware Modality Connector for Speech Language Modelscited by 16
  26. 2023Leveraging Large Language Models for Exploiting ASR Uncertaintycited by 15
  27. 2025Audio Jailbreak: An Open Comprehensive Benchmark for Jailbreaking Large Audio-Language Modelscited by 15
  28. 2024Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversationcited by 14
  29. 2024Speech ReaLLM -- Real-time Streaming Speech Recognition with Multimodal LLMs by Teaching the Flow of Timecited by 14
  30. 2025SIFT-50M: A Large-Scale Multilingual Dataset for Speech Instruction Fine-Tuningcited by 14
  31. 2024Listen Again and Choose the Right Answer: A New Paradigm for Automatic Speech Recognition with Large Language Modelscited by 13
  32. 2024PSLM: Parallel Generation of Text and Speech with LLMs for Low-Latency Spoken Dialogue Systemscited by 13
  33. 2024On Pre-training of Multimodal Language Models Customized for Chart Understandingcited by 13
  34. 2024Enabling Auditory Large Language Models for Automatic Speech Quality Evaluationcited by 13
  35. 2024Decoder-only Architecture for Streaming End-to-end Speech Recognitioncited by 12
  36. 2025Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal LLMscited by 11
  37. 2023Integrating Pre-Trained Speech and Language Models for End-to-End Speech Recognitioncited by 10
  38. 2024DiscreteSLU: A Large Language Model with Self-Supervised Discrete Speech Units for Spoken Language Understandingcited by 10

50 most cited of 139 All 139 (JSON)