# Stateful Conformer with Cache-based Inference for Streaming Automatic Speech Recognition

arXiv: https://arxiv.org/abs/2312.17279 · HTML: https://mlsanity.com/paper/2312.17279/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 35 references that are in the map.
- First posted on arXiv 27 Dec 2023.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio](https://mlsanity.com/paper/2106.06909/) (2021)
- [NeMo: a toolkit for building AI applications using Neural Modules](https://mlsanity.com/paper/1909.09577/) (2019)
- [Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss](https://mlsanity.com/paper/2002.02562/) (2020)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2102.01547/) (2021)
- [Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition](https://mlsanity.com/paper/2305.05084/) (2023)
- [Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognition](https://mlsanity.com/paper/2010.10759/) (2020)
- [Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Dataset](https://mlsanity.com/paper/2010.11395/) (2020)
- [Streaming automatic speech recognition with the transformer model](https://mlsanity.com/paper/2001.02674/) (2020)
- [FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularization](https://mlsanity.com/paper/2010.11148/) (2020)
- [A Better and Faster End-to-End Model for Streaming ASR](https://mlsanity.com/paper/2011.10798/) (2020)
- [Earnings-22: A Practical Benchmark for Accents in the Wild](https://mlsanity.com/paper/2203.15591/) (2022)
- [Streaming Transformer-based Acoustic Models Using Self-attention with Augmented Memory](https://mlsanity.com/paper/2005.08042/) (2020)
- [Synchronous Transformers for End-to-End Speech Recognition](https://mlsanity.com/paper/1912.02958/) (2019)
- [Low Latency End-to-End Streaming Speech Recognition with a Scout Network](https://mlsanity.com/paper/2003.10369/) (2020)
- [Enhancing Monotonic Multihead Attention for Streaming ASR](https://mlsanity.com/paper/2005.09394/) (2020)
- [Multi-mode Transformer Transducer with Stochastic Future Context](https://mlsanity.com/paper/2106.09760/) (2021)

## Built on by (most cited first, 25 of 25)

- [Moshi: a speech-text foundation model for real-time dialogue](https://mlsanity.com/paper/2410.00037/) (2024)
- [Fast Context-Biasing for CTC and Transducer ASR models with CTC-based Word Spotter](https://mlsanity.com/paper/2406.07096/) (2024)
- [Voxtral Realtime](https://mlsanity.com/paper/2602.11298/) (2026)
- [The Silent Thought: Modeling Internal Cognition in Full-Duplex Spoken Dialogue Models via Latent Reasoning](https://mlsanity.com/paper/2603.17837/) (2026)
- [Open Universal Arabic ASR Leaderboard](https://mlsanity.com/paper/2412.13788/) (2024)
- [RASMALAI: Resources for Adaptive Speech Modeling in Indian Languages with Accents and Intonations](https://mlsanity.com/paper/2505.18609/) (2025)
- [TurboBias: Universal ASR Context-Biasing powered by GPU-accelerated Phrase-Boosting Tree](https://mlsanity.com/paper/2508.07014/) (2025)
- [On the Robust Approximation of ASR Metrics](https://mlsanity.com/paper/2502.12408/) (2025)
- [NIRANTAR: Continual Learning with New Languages and Domains on Real-world Speech Data](https://mlsanity.com/paper/2507.00534/) (2025)
- [Open Source State-Of-the-Art Solution for Romanian Speech Recognition](https://mlsanity.com/paper/2511.03361/) (2025)
- [NIM4-ASR: Towards Efficient, Robust, and Customizable Real-Time LLM-Based ASR](https://mlsanity.com/paper/2604.18105/) (2026)
- [Reducing the Offline-Streaming Gap for Unified ASR Transducer with Consistency Regularization](https://mlsanity.com/paper/2604.19079/) (2026)
- [Gradient-Based Speech-to-Text Alignment for Any ASR Model: From CTC to Speech LLMs](https://mlsanity.com/paper/2607.06831/) (2026)
- [Enabling Streaming User Transcription in Full-Duplex Speech-to-Speech Models](https://mlsanity.com/paper/2609.15759/) (2026)
- [Chunk-wise Attention Transducers for Fast and Accurate Streaming Speech-to-Text](https://mlsanity.com/paper/2602.24245/) (2026)
- [Data Scale, Not Latency, Shapes Cross-Lingual Encoder Transfer in Streaming ASR](https://mlsanity.com/paper/2606.24169/) (2026)
- [From a Multilingual Streaming ASR Backbone to Kenyan-Language Systems: Data-Centric Adaptation of Nemotron 3.5 for Kikuyu, Dholuo, and Kalenjin](https://mlsanity.com/paper/2607.18912/) (2026)
- [StreamHear: Domain-Adapted Pseudo-Labeling for Semi-Supervised Streaming Speech Recognition](https://mlsanity.com/paper/2608.13717/) (2026)
- [Typhoon ASR Streaming: Steerable Low-Latency Thai Speech Recognition with Real-Time Shallow Fusion](https://mlsanity.com/paper/2609.14991/) (2026)
- [Voice-Light: A Full-Duplex Cascaded Voice Agent with Causal Turn-Taking and Speculative Generation](https://mlsanity.com/paper/2609.20995/) (2026)
- [All In Good Time: Causality-Aware Framework for LLM-Based Simultaneous Speech-to-Speech Translation](https://mlsanity.com/paper/2609.30416/) (2026)
- [Factorized Delayed Streams Modeling for LLM-based Streaming ASR](https://mlsanity.com/paper/2610.04333/) (2026)
- [Mind the Accent Gap: British Accent Robustness in Speech-Driven Financial Voice Assistants](https://mlsanity.com/paper/2610.06587/) (2026)
- [Jarvis: A Proactive Speech Agent for Multi-Party Conversations](https://mlsanity.com/paper/2610.07506/) (2026)
- [O'ido: Open-Vocabulary Speech Recognition on a 5 Microcontroller without a Neural Accelerator](https://mlsanity.com/paper/2610.13447/) (2026)

## Abstract (arXiv)

In this paper, we propose an efficient and accurate streaming speech recognition model based on the FastConformer architecture. We adapted the FastConformer architecture for streaming applications through: (1) constraining both the look-ahead and past contexts in the encoder, and (2) introducing an activation caching mechanism to enable the non-autoregressive encoder to operate autoregressively during inference. The proposed model is thoughtfully designed in a way to eliminate the accuracy disparity between the train and inference time which is common for many streaming models. Furthermore, our proposed encoder works with various decoder configurations including Connectionist Temporal Classification (CTC) and RNN-Transducer (RNNT) decoders. Additionally, we introduced a hybrid CTC/RNNT architecture which utilizes a shared encoder with both a CTC and RNNT decoder to boost the accuracy and save computation. We evaluate the proposed model on LibriSpeech dataset and a multi-domain large scale dataset and demonstrate that it can achieve better accuracy with lower latency and inference time compared to a conventional buffered streaming model baseline. We also showed that training a model with multiple latencies can achieve better accuracy than single latency models while it enables us to support multiple latencies with a single model. Our experiments also showed the hybrid architecture would not only speedup the convergence of the CTC decoder but also improves the accuracy of streaming models compared to single decoder models.