# Speculative Speech Recognition by Audio-Prefixed Low-Rank Adaptation of Language Models

arXiv: https://arxiv.org/abs/2407.04641 · HTML: https://mlsanity.com/paper/2407.04641/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 33 references that are in the map.
- First posted on arXiv 5 Jul 2024.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss](https://mlsanity.com/paper/2002.02562/) (2020)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation](https://mlsanity.com/paper/1805.04699/) (2018)
- [SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network](https://mlsanity.com/paper/2104.02133/) (2021)
- [FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularization](https://mlsanity.com/paper/2010.11148/) (2020)
- [A Better and Faster End-to-End Model for Streaming ASR](https://mlsanity.com/paper/2011.10798/) (2020)
- [Two-Pass End-to-End Speech Recognition](https://mlsanity.com/paper/1908.10992/) (2019)
- [Towards Fast and Accurate Streaming End-to-End ASR](https://mlsanity.com/paper/2004.11544/) (2020)
- [Multitask Training with Text Data for End-to-End Speech Recognition](https://mlsanity.com/paper/2010.14318/) (2020)
- [USTED: Improving ASR with a Unified Speech and Text Encoder-Decoder](https://mlsanity.com/paper/2202.06045/) (2022)
- [Lookahead When It Matters: Adaptive Non-causal Transformers for Streaming Neural Transducers](https://mlsanity.com/paper/2305.04159/) (2023)
- [Personalized Predictive ASR for Latency Reduction in Voice Assistants](https://mlsanity.com/paper/2305.13794/) (2023)
- [Statistical Testing on ASR Performance via Blockwise Bootstrap](https://mlsanity.com/paper/1912.09508/) (2019)
- [On-the-fly Text Retrieval for End-to-End ASR Adaptation](https://mlsanity.com/paper/2303.10942/) (2023)

## Built on by (most cited first, 3 of 3)

- [Low-Rank Adaptation for Foundation Models: A Comprehensive Review](https://mlsanity.com/paper/2501.00365/) (2024)
- [Predictive Speech Recognition and End-of-Utterance Detection Towards Spoken Dialog Systems](https://mlsanity.com/paper/2409.19990/) (2024)
- [AS2D: Accelerating On-Demand Audio Understanding on Mobile Devices](https://mlsanity.com/paper/2609.37617/) (2026)

## Abstract (arXiv)

This paper explores speculative speech recognition (SSR), where we empower conventional automatic speech recognition (ASR) with speculation capabilities, allowing the recognizer to run ahead of audio. We introduce a metric for measuring SSR performance and we propose a model which does SSR by combining a RNN-Transducer-based ASR system with an audio-prefixed language model (LM). The ASR system transcribes ongoing audio and feeds the resulting transcripts, along with an audio-dependent prefix, to the LM, which speculates likely completions for the transcriptions. We experiment with a variety of ASR datasets on which show the efficacy our method and the feasibility of SSR as a method of reducing ASR latency.