# Decoder-only Architecture for Streaming End-to-end Speech Recognition arXiv: https://arxiv.org/abs/2406.16107 ยท HTML: https://mlsanity.com/paper/2406.16107/ ## Key facts - Cited by 13 papers in the map (data through 14 Oct 2026). - Builds on 28 of its 33 references that are in the map. - First posted on arXiv 23 Jun 2024. ## Builds on (most cited first, 28 of 28) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022) - [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013) - [SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities](https://mlsanity.com/paper/2305.11000/) (2023) - [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://mlsanity.com/paper/1512.02595/) (2015) - [AudioPaLM: A Large Language Model That Can Speak and Listen](https://mlsanity.com/paper/2306.12925/) (2023) - [Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss](https://mlsanity.com/paper/2002.02562/) (2020) - [On decoder-only architecture for speech-to-text and large language model integration](https://mlsanity.com/paper/2307.03917/) (2023) - [Prompting Large Language Models with Speech Recognition Abilities](https://mlsanity.com/paper/2307.11795/) (2023) - [WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2102.01547/) (2021) - [Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognition](https://mlsanity.com/paper/2010.10759/) (2020) - [Exploring Architectures, Data and Units For Streaming End-to-End Speech Recognition with RNN-Transducer](https://mlsanity.com/paper/1801.00841/) (2018) - [CTC-Segmentation of Large Corpora for German End-to-end Speech Recognition](https://mlsanity.com/paper/2007.09127/) (2020) - [EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decoding](https://mlsanity.com/paper/1507.08240/) (2015) - [Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation tasks](https://mlsanity.com/paper/2309.07937/) (2023) - [FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularization](https://mlsanity.com/paper/2010.11148/) (2020) - [Two-Pass End-to-End Speech Recognition](https://mlsanity.com/paper/1908.10992/) (2019) - [Self-Attention Aligner: A Latency-Control End-to-End Model for ASR Using Self-Attention Network and Chunk-Hopping](https://mlsanity.com/paper/1902.06450/) (2019) - [SpeechPrompt v2: Prompt Tuning for Speech Classification Tasks](https://mlsanity.com/paper/2303.00733/) (2023) - [Transformer ASR with Contextual Block Processing](https://mlsanity.com/paper/1910.07204/) (2019) - [Phoneme Based Neural Transducer for Large Vocabulary Speech Recognition](https://mlsanity.com/paper/2010.16368/) (2020) - [Decoder-only Architecture for Speech Recognition with CTC Prompts and Text Data Augmentation](https://mlsanity.com/paper/2309.08876/) (2023) - [Semi-Autoregressive Streaming ASR With Label Context](https://mlsanity.com/paper/2309.10926/) (2023) - [BASS: Block-wise Adaptation for Speech Summarization](https://mlsanity.com/paper/2307.08217/) (2023) - [Transformer-based Streaming ASR with Cumulative Attention](https://mlsanity.com/paper/2203.05736/) (2022) - [Lego-Features: Exporting modular encoder features for streaming and deliberation ASR](https://mlsanity.com/paper/2304.00173/) (2023) - [Integration of Frame- and Label-synchronous Beam Search for Streaming Encoder-decoder Speech Recognition](https://mlsanity.com/paper/2307.12767/) (2023) - [Integrating Pretrained ASR and LM to Perform Sequence Generation for Spoken Language Understanding](https://mlsanity.com/paper/2307.11005/) (2023) ## Built on by (most cited first, 13 of 13) - [On The Landscape of Spoken Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2504.08528/) (2025) - [Language Model Can Listen While Speaking](https://mlsanity.com/paper/2408.02622/) (2024) - [BESTOW: Efficient and Streamable Speech Language Model with the Best of Two Worlds in GPT and T5](https://mlsanity.com/paper/2406.19954/) (2024) - [Efficient Streaming LLM for Speech Recognition](https://mlsanity.com/paper/2410.03752/) (2024) - [Mitigating Sycophancy in Decoder-Only Transformer Architectures: Synthetic Data Intervention](https://mlsanity.com/paper/2411.10156/) (2024) - [LauraTSE: Target Speaker Extraction using Auto-Regressive Decoder-Only Language Models](https://mlsanity.com/paper/2504.07402/) (2025) - [Transducer-Llama: Integrating LLMs into Streamable Transducer-based Speech Recognition](https://mlsanity.com/paper/2412.16464/) (2024) - [CJST: CTC Compressor based Joint Speech and Text Training for Decoder-Only ASR](https://mlsanity.com/paper/2411.07607/) (2024) - [MFLA: Monotonic Finite Look-ahead Attention for Streaming Speech Recognition](https://mlsanity.com/paper/2506.03722/) (2025) - [WhisperRT -- Turning Whisper into a Causal Streaming Model](https://mlsanity.com/paper/2508.12301/) (2025) - [Spiralformer: Low Latency Encoder for Streaming Speech Recognition with Circular Layer Skipping and Early Exiting](https://mlsanity.com/paper/2510.00982/) (2025) - [Do we really need Self-Attention for Streaming Automatic Speech Recognition?](https://mlsanity.com/paper/2601.19960/) (2026) - [Alignment-Path Distillation from Non-streaming ASR-LLMs for Streaming Speech Recognition](https://mlsanity.com/paper/2609.20121/) (2026) ## Abstract (arXiv) Decoder-only language models (LMs) have been successfully adopted for speech-processing tasks including automatic speech recognition (ASR). The LMs have ample expressiveness and perform efficiently. This efficiency is a suitable characteristic for streaming applications of ASR. In this work, we propose to use a decoder-only architecture for blockwise streaming ASR. In our approach, speech features are compressed using CTC output and context embedding using blockwise speech subnetwork, and are sequentially provided as prompts to the decoder. The decoder estimates the output tokens promptly at each block. To this end, we also propose a novel training scheme using random-length prefix prompts to make the model robust to the truncated prompts caused by blockwise processing. An experimental comparison shows that our proposed decoder-only streaming ASR achieves 8% relative word error rate reduction in the LibriSpeech test-other set while being twice as fast as the baseline model.