mlsanityData to 14 Oct 2026

Speech ReaLLM -- Real-time Streaming Speech Recognition with Multimodal LLMs by Teaching the Flow of Time

arXiv 2406.09569 ↗First posted 13 Jun 2024cs.CL cs.AI cs.SDcited by 15

Not on the map. Why?

Abstract (arXiv)

We introduce Speech ReaLLM, a new ASR architecture that marries "decoder-only" ASR with the RNN-T to make multimodal LLM architectures capable of real-time streaming. This is the first "decoder-only" ASR architecture designed to handle continuous audio without explicit end-pointing. Speech ReaLLM is a special case of the more general ReaLLM ("real-time LLM") approach, also introduced here for the first time. The idea is inspired by RNN-T: Instead of generating a response only at the end of a user prompt, generate after every input token received in real time (it is often empty). On Librispeech "test", an 80M Speech ReaLLM achieves WERs of 3.0% and 7.4% in real time (without an external LM or auxiliary loss). This is only slightly above a 3x larger Attention-Encoder-Decoder baseline. We also show that this way, an LLM architecture can learn to represent and reproduce the flow of time; and that a pre-trained 7B LLM can be fine-tuned to do reasonably well on this task.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 15 of its 20 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

15 of 20 references.

References · 15

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  4. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  5. 2018Self-Attention with Relative Position Representationscited by 1,120
  6. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 187
  7. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 145
  8. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 140
  9. 2023SeamlessM4T: Massively Multilingual & Multimodal Machine Translationcited by 125
  10. 2020Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognitioncited by 113
  11. 2019Monotonic Infinite Lookback Attention for Simultaneous Machine Translationcited by 104
  12. 2017Online and Linear-Time Attention by Enforcing Monotonic Alignmentscited by 85
3 more
  1. 2017Monotonic Chunkwise Attentioncited by 83
  2. 2023Efficient Monotonic Multihead Attentioncited by 8
  3. 2021Streaming Transformer Transducer Based Speech Recognition Using Non-Causal Convolutioncited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 3; 2026: 4.

15 papers.

Sort:

By area:

  1. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 173
  2. 2024Language Model Can Listen While Speakingcited by 63
  3. 2024SocialMind: LLM-based Proactive AR Social Assistive System with Human-like Perception for In-situ Live Interactionscited by 25
  4. 2024BESTOW: Efficient and Streamable Speech Language Model with the Best of Two Worlds in GPT and T5cited by 19
  5. 2024Efficient Streaming LLM for Speech Recognitioncited by 12
  6. 2025Word Level Timestamp Generation for Automatic Speech Recognition and Translationcited by 6
  7. 2026Uni-ASR: Unified LLM-Based Architecture for Non-Streaming and Streaming Automatic Speech Recognitioncited by 6
  8. 2024Overcoming Non-monotonicity in Transducer-based Streaming Generationcited by 4
  9. 2024Transducer-Llama: Integrating LLMs into Streamable Transducer-based Speech Recognitioncited by 4
  10. 2024Optimizing Contextual Speech Recognition Using Vector Quantization for Efficient Retrievalcited by 3
  11. 2025SimulS2S-LLM: Unlocking Simultaneous Inference of Speech LLMs for Speech-to-Speech Translationcited by 3
  12. 2026From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Modelscited by 3
3 more
  1. 2025PAC: Pronunciation-Aware Contextualized Large Language Model-based Automatic Speech Recognitioncited by 1
  2. 2026TRADE: Transducer-Augmented Decoder for Speech LLMcited by 0
  3. 2026Alignment-Path Distillation from Non-streaming ASR-LLMs for Streaming Speech Recognitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Bridging the gap between streaming and non-streaming ASR systems bydistilling ensembles of CTC and RNN-T models
  2. 2024Joint Optimization of Streaming and Non-Streaming Automatic Speech Recognition with Multi-Decoder and Knowledge Distillation
  3. 2024Fast Streaming Transducer ASR Prototyping via Knowledge Distillation with Whisper
  4. 2022Two-Pass End-to-End ASR Model Compression
  5. 2024Internalizing ASR with Implicit Chain of Thought for Efficient Speech-to-Speech Conversational LLM
  6. 2026NIM4-ASR: Towards Efficient, Robust, and Customizable Real-Time LLM-Based ASR
  7. 2026dLLM-ASR: A Faster Diffusion LLM-based Framework for Speech Recognition
  8. 2026MURMUR: An Efficient Inference System for Long-Form ASR