mlsanityData to 7 Oct 2026

Efficient Streaming LLM for Speech Recognition

arXiv 2410.03752 ↗First posted 2 Oct 2024cs.SD cs.AI cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

Recent works have shown that prompting large language models with audio encodings can unlock speech recognition capabilities. However, existing techniques do not scale efficiently, especially while handling long form streaming audio inputs -- not only do they extrapolate poorly beyond the audio length seen during training, but they are also computationally inefficient due to the quadratic cost of attention. In this work, we introduce SpeechLLM-XL, a linear scaling decoder-only model for streaming speech recognition. We process audios in configurable chunks using limited attention window for reduced computation, and the text tokens for each audio chunk are generated auto-regressively until an EOS is predicted. During training, the transcript is segmented into chunks, using a CTC forced alignment estimated from encoder output. SpeechLLM-XL with 1.28 seconds chunk size achieves 2.7%/6.7% WER on LibriSpeech test clean/other, and it shows no quality degradation on long form utterances 10x longer than the training utterances.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 31 of its 36 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

31 of 36 references.

References · 31

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  6. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  8. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  9. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  10. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
  11. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  12. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
19 more
  1. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 194
  2. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 144
  3. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 139
  4. 2023SeamlessM4T: Massively Multilingual & Multimodal Machine Translationcited by 124
  5. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  6. 2017Monotonic Chunkwise Attentioncited by 82
  7. 2023Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation taskscited by 71
  8. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  9. 2023AudioChatLlama: Towards General-Purpose Speech Abilities for LLMscited by 59
  10. 2024Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Lengthcited by 46
  11. 2023SpeechPrompt v2: Prompt Tuning for Speech Classification Taskscited by 31
  12. 2023End-to-End Speech Recognition Contextualization with Large Language Modelscited by 22
  13. 2019From Senones to Chenones: Tied Context-Dependent Graphemes for Hybrid Speech Recognitioncited by 20
  14. 2024Speech ReaLLM -- Real-time Streaming Speech Recognition with Multimodal LLMs by Teaching the Flow of Timecited by 14
  15. 2024Decoder-only Architecture for Streaming End-to-end Speech Recognitioncited by 12
  16. 2020Online Automatic Speech Recognition with Listen, Attend and Spell Modelcited by 10
  17. 2023Chunked Attention-based Encoder-Decoder Model for Streaming Speech Recognitioncited by 7
  18. 2022Monotonic segmental attention for automatic speech recognitioncited by 4
  19. 2023Integrating Pretrained ASR and LM to Perform Sequence Generation for Spoken Language Understandingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 5.

10 papers.

  1. 2025Evaluation of LLMs in Speech is Often Flawed: Test Set Contamination in Large Language Models for Speech Recognitioncited by 9
  2. 2026Uni-ASR: Unified LLM-Based Architecture for Non-Streaming and Streaming Automatic Speech Recognitioncited by 5
  3. 2025MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognitioncited by 4
  4. 2026Whisper-CD: Accurate Long-Form Speech Recognition using Multi-Negative Contrastive Decodingcited by 3
  5. 2025MFLA: Monotonic Finite Look-ahead Attention for Streaming Speech Recognitioncited by 1
  6. 2025PAC: Pronunciation-Aware Contextualized Large Language Model-based Automatic Speech Recognitioncited by 1
  7. 2026Adaptive Turn-Taking for Real-time Multi-Party Voice Agentscited by 1
  8. 2025WhisperRT -- Turning Whisper into a Causal Streaming Modelcited by 0
  9. 2026Decoder-only Conformer with Modality-aware Sparse Mixtures of Experts for ASRcited by 0
  10. 2026Forewarned is Forearmed: When Non-Sequential Embedding Turns Into an Anomaly Detectorcited by 0