# Understanding Zero-shot Rare Word Recognition Improvements Through LLM Integration

arXiv: https://arxiv.org/abs/2502.16142 · HTML: https://mlsanity.com/paper/2502.16142/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 13 references that are in the map.
- First posted on arXiv 22 Feb 2025.

## Builds on (most cited first, 10 of 10)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [An Embarrassingly Simple Approach for LLM with Strong ASR Capacity](https://mlsanity.com/paper/2402.08846/) (2024)
- [Zipformer: A faster and better encoder for automatic speech recognition](https://mlsanity.com/paper/2310.11230/) (2023)
- [PriMock57: A Dataset Of Primary Care Mock Consultations](https://mlsanity.com/paper/2204.00333/) (2022)
- [CTC-Assisted LLM-Based Contextual ASR](https://mlsanity.com/paper/2411.06437/) (2024)
- [Multi-stage Large Language Model Correction for Speech Recognition](https://mlsanity.com/paper/2310.11532/) (2023)
- [Exploring the Integration of Large Language Models into Automatic Speech Recognition Systems: An Empirical Study](https://mlsanity.com/paper/2307.06530/) (2023)
- [ed-cec: improving rare word recognition using asr postprocessing based on error detection and context-aware error correction](https://mlsanity.com/paper/2310.05129/) (2023)
- [Meta-Learning for improving rare word recognition in end-to-end ASR](https://mlsanity.com/paper/2102.12624/) (2021)

## Abstract (arXiv)

In this study, we investigate the integration of a large language model (LLM) with an automatic speech recognition (ASR) system, specifically focusing on enhancing rare word recognition performance. Using a 190,000-hour dataset primarily sourced from YouTube, pre-processed with Whisper V3 pseudo-labeling, we demonstrate that the LLM-ASR architecture outperforms traditional Zipformer-Transducer models in the zero-shot rare word recognition task, after training on a large dataset. Our analysis reveals that the LLM contributes significantly to improvements in rare word error rate (R-WER), while the speech encoder primarily determines overall transcription performance (Orthographic Word Error Rate, O-WER, and Normalized Word Error Rate, N-WER). Through extensive ablation studies, we highlight the importance of adapter integration in aligning speech encoder outputs with the LLM's linguistic capabilities. Furthermore, we emphasize the critical role of high-quality labeled data in achieving optimal performance. These findings provide valuable insights into the synergy between LLM-based ASR architectures, paving the way for future advancements in large-scale LLM-based speech recognition systems.