# A First Look at Bugs in LLM Inference Engines

arXiv: https://arxiv.org/abs/2506.09713 · HTML: https://mlsanity.com/paper/2506.09713/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jun 2025.

## Built on by (most cited first, 5 of 5)

- [Real Faults in Model Context Protocol (MCP) Software: a Comprehensive Taxonomy](https://mlsanity.com/paper/2603.05637/) (2026)
- [Understanding Bugs in Modern Agentic Frameworks: A Study of Symptoms, Root Causes, and Triggering Conditions](https://mlsanity.com/paper/2604.08906/) (2026)
- [Continuous Discovery of Vulnerabilities in LLM Serving Systems with Fuzzing](https://mlsanity.com/paper/2605.11202/) (2026)
- [Ekka: Automated Diagnosis of Silent Errors in LLM Inference](https://mlsanity.com/paper/2606.04594/) (2026)
- [Reliability Testing of Medical Model Performance under Distributed Deployment](https://mlsanity.com/paper/2609.36525/) (2026)

## Abstract (arXiv)

Large language model-specific inference engines (in short as LLM inference engines) have become a fundamental component of modern AI infrastructure, enabling the deployment of LLM-powered applications (LLM apps) across cloud and local devices. Despite their critical role, LLM inference engines are prone to bugs due to the immense resource demands of LLMs and the complexities of cross-platform compatibility. However, a systematic understanding of these bugs remains lacking. To bridge this gap, we present the first empirical study on bugs in LLM inference engines. We mine official repositories of 5 widely adopted LLM inference engines, constructing a comprehensive dataset of 929 real-world bugs. Through a rigorous open coding process, we analyze these bugs to uncover their symptoms, root causes, commonality, fix effort, fix strategies, and temporal evolution. Our findings reveal six bug symptom types and a taxonomy of 28 root causes, shedding light on the key challenges in bug detection and location within LLM inference engines. Based on these insights, we propose a series of actionable implications for researchers, inference engine vendors, and LLM app developers, along with general guidelines for developing LLM inference engines.