# Exploring the Integration of Large Language Models into Automatic Speech Recognition Systems: An Empirical Study arXiv: https://arxiv.org/abs/2307.06530 ยท HTML: https://mlsanity.com/paper/2307.06530/ ## Key facts - Cited by 7 papers in the map (data through 14 Oct 2026). - Builds on 28 of its 34 references that are in the map. - First posted on arXiv 13 Jul 2023. ## Builds on (most cited first, 28 of 28) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022) - [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023) - [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022) - [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022) - [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020) - [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022) - [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013) - [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018) - [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015) - [Attention-Based Models for Speech Recognition](https://mlsanity.com/paper/1506.07503/) (2015) - [AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baseline](https://mlsanity.com/paper/1709.05522/) (2017) - [Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning](https://mlsanity.com/paper/1609.06773/) (2016) - [WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2102.01547/) (2021) - [An analysis of incorporating an external language model into a sequence-to-sequence model](https://mlsanity.com/paper/1712.01996/) (2017) - [ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context](https://mlsanity.com/paper/2005.03191/) (2020) - [Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understanding](https://mlsanity.com/paper/2207.02971/) (2022) - [Two-Pass End-to-End Speech Recognition](https://mlsanity.com/paper/1908.10992/) (2019) - [Listening while Speaking: Speech Chain by Deep Learning](https://mlsanity.com/paper/1707.04879/) (2017) - [Distilling the Knowledge of BERT for Sequence-to-Sequence ASR](https://mlsanity.com/paper/2008.03822/) (2020) - [Unified Streaming and Non-streaming Two-pass End-to-end Model for Speech Recognition](https://mlsanity.com/paper/2012.05481/) (2020) - [RescoreBERT: Discriminative Speech Recognition Rescoring with BERT](https://mlsanity.com/paper/2202.01094/) (2022) - [Neural Speech Recognizer: Acoustic-to-Word LSTM Model for Large Vocabulary Speech Recognition](https://mlsanity.com/paper/1610.09975/) (2016) - [Innovative Bert-based Reranking Language Models for Speech Recognition](https://mlsanity.com/paper/2104.04950/) (2021) - [Effect and Analysis of Large-scale Language Model Rescoring on Competitive ASR Systems](https://mlsanity.com/paper/2204.00212/) (2022) - [Knowledge Transfer from Large-scale Pretrained Language Models to End-to-end Speech Recognizers](https://mlsanity.com/paper/2202.07894/) (2022) - [Improving Rare Word Recognition with LM-aware MWER Training](https://mlsanity.com/paper/2204.07553/) (2022) ## Built on by (most cited first, 7 of 7) - [Can Whisper perform speech-based in-context learning?](https://mlsanity.com/paper/2309.07081/) (2023) - [Fewer Hallucinations, More Verification: A Three-Stage LLM-Based Framework for ASR Error Correction](https://mlsanity.com/paper/2505.24347/) (2025) - [Towards interfacing large language models with ASR systems using confidence measures and prompting](https://mlsanity.com/paper/2407.21414/) (2024) - [Multiple Representation Transfer from Large Language Models to End-to-End ASR Systems](https://mlsanity.com/paper/2309.04031/) (2023) - [Revisiting N-Gram Models: Their Impact in Modern Neural Networks for Handwritten Text Recognition](https://mlsanity.com/paper/2404.19317/) (2024) - [Understanding Zero-shot Rare Word Recognition Improvements Through LLM Integration](https://mlsanity.com/paper/2502.16142/) (2025) - [RECOVER: Robust Entity Correction via agentic Orchestration of hypothesis Variants for Evidence-based Recovery](https://mlsanity.com/paper/2603.16411/) (2026) ## Abstract (arXiv) This paper explores the integration of Large Language Models (LLMs) into Automatic Speech Recognition (ASR) systems to improve transcription accuracy. The increasing sophistication of LLMs, with their in-context learning capabilities and instruction-following behavior, has drawn significant attention in the field of Natural Language Processing (NLP). Our primary focus is to investigate the potential of using an LLM's in-context learning capabilities to enhance the performance of ASR systems, which currently face challenges such as ambient noise, speaker accents, and complex linguistic contexts. We designed a study using the Aishell-1 and LibriSpeech datasets, with ChatGPT and GPT-4 serving as benchmarks for LLM capabilities. Unfortunately, our initial experiments did not yield promising results, indicating the complexity of leveraging LLM's in-context learning for ASR applications. Despite further exploration with varied settings and models, the corrected sentences from the LLMs frequently resulted in higher Word Error Rates (WER), demonstrating the limitations of LLMs in speech applications. This paper provides a detailed overview of these experiments, their results, and implications, establishing that using LLMs' in-context learning capabilities to correct potential errors in speech recognition transcriptions is still a challenging task at the current stage.