mlsanityData to 7 Oct 2026

Decoder-only Architecture for Streaming End-to-end Speech Recognition

arXiv 2406.16107 ↗First posted 23 Jun 2024eess.AS cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

Decoder-only language models (LMs) have been successfully adopted for speech-processing tasks including automatic speech recognition (ASR). The LMs have ample expressiveness and perform efficiently. This efficiency is a suitable characteristic for streaming applications of ASR. In this work, we propose to use a decoder-only architecture for blockwise streaming ASR. In our approach, speech features are compressed using CTC output and context embedding using blockwise speech subnetwork, and are sequentially provided as prompts to the decoder. The decoder estimates the output tokens promptly at each block. To this end, we also propose a novel training scheme using random-length prefix prompts to make the model robust to the truncated prompts caused by blockwise processing. An experimental comparison shows that our proposed decoder-only streaming ASR achieves 8% relative word error rate reduction in the LibriSpeech test-other set while being twice as fast as the baseline model.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 28 of its 33 references that are in the map. First posted on arXiv 23 Jun 2024.

Builds on

28 of 33 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 666
  4. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  5. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 449
  6. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  7. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 232
  8. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 144
  9. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 139
  10. 2021WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkitcited by 123
  11. 2020Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognitioncited by 111
  12. 2018Exploring Architectures, Data and Units For Streaming End-to-End Speech Recognition with RNN-Transducercited by 95
16 more
  1. 2020CTC-Segmentation of Large Corpora for German End-to-end Speech Recognitioncited by 75
  2. 2015EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decodingcited by 73
  3. 2023Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation taskscited by 71
  4. 2020FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularizationcited by 66
  5. 2019Two-Pass End-to-End Speech Recognitioncited by 62
  6. 2019Self-Attention Aligner: A Latency-Control End-to-End Model for ASR Using Self-Attention Network and Chunk-Hoppingcited by 33
  7. 2023SpeechPrompt v2: Prompt Tuning for Speech Classification Taskscited by 31
  8. 2019Transformer ASR with Contextual Block Processingcited by 28
  9. 2020Phoneme Based Neural Transducer for Large Vocabulary Speech Recognitioncited by 20
  10. 2023Decoder-only Architecture for Speech Recognition with CTC Prompts and Text Data Augmentationcited by 10
  11. 2023Semi-Autoregressive Streaming ASR With Label Contextcited by 5
  12. 2023BASS: Block-wise Adaptation for Speech Summarizationcited by 4
  13. 2022Transformer-based Streaming ASR with Cumulative Attentioncited by 3
  14. 2023Lego-Features: Exporting modular encoder features for streaming and deliberation ASRcited by 3
  15. 2023Integration of Frame- and Label-synchronous Beam Search for Streaming Encoder-decoder Speech Recognitioncited by 3
  16. 2023Integrating Pretrained ASR and LM to Perform Sequence Generation for Spoken Language Understandingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 5; 2026: 1.

12 papers.

  1. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  2. 2024Language Model Can Listen While Speakingcited by 62
  3. 2024BESTOW: Efficient and Streamable Speech Language Model with the Best of Two Worlds in GPT and T5cited by 18
  4. 2024Efficient Streaming LLM for Speech Recognitioncited by 10
  5. 2024Mitigating Sycophancy in Decoder-Only Transformer Architectures: Synthetic Data Interventioncited by 5
  6. 2025LauraTSE: Target Speaker Extraction using Auto-Regressive Decoder-Only Language Modelscited by 5
  7. 2024Transducer-Llama: Integrating LLMs into Streamable Transducer-based Speech Recognitioncited by 4
  8. 2024CJST: CTC Compressor based Joint Speech and Text Training for Decoder-Only ASRcited by 2
  9. 2025MFLA: Monotonic Finite Look-ahead Attention for Streaming Speech Recognitioncited by 1
  10. 2025WhisperRT -- Turning Whisper into a Causal Streaming Modelcited by 0
  11. 2025Spiralformer: Low Latency Encoder for Streaming Speech Recognition with Circular Layer Skipping and Early Exitingcited by 0
  12. 2026Do we really need Self-Attention for Streaming Automatic Speech Recognition?cited by 0