# An Empirical Study on Challenges for LLM Application Developers

arXiv: https://arxiv.org/abs/2408.05002 · HTML: https://mlsanity.com/paper/2408.05002/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Aug 2024.

## Built on by (most cited first, 18 of 18)

- [Defining and Detecting the Defects of the Large Language Model-based Autonomous Agents](https://mlsanity.com/paper/2412.18371/) (2024)
- [Large Language Models (LLMs) for Source Code Analysis: applications, models and datasets](https://mlsanity.com/paper/2503.17502/) (2025)
- [From REST to MCP: An Empirical Study of API Wrapping and Automated Server Generation for LLM Agents](https://mlsanity.com/paper/2507.16044/) (2025)
- [Small Models, Big Tasks: An Exploratory Empirical Study on Small Language Models for Function Calling](https://mlsanity.com/paper/2504.19277/) (2025)
- [Walking the Tightrope of LLMs for Software Development: A Practitioners' Perspective](https://mlsanity.com/paper/2511.06428/) (2025)
- [Breaking Minds, Breaking Systems: Jailbreaking Large Language Models via Human-like Psychological Manipulation](https://mlsanity.com/paper/2512.18244/) (2025)
- [How Do Practitioners Build SE Agents? Insights from a Mixed-Methods Study](https://mlsanity.com/paper/2607.10856/) (2026)
- [Understanding and supporting how developers prompt for LLM-powered code editing in practice](https://mlsanity.com/paper/2504.20196/) (2025)
- [An Empirical Study of OpenAI API Discussions on Stack Overflow](https://mlsanity.com/paper/2505.04084/) (2025)
- [Key Considerations for Domain Expert Involvement in LLM Design and Evaluation: An Ethnographic Study](https://mlsanity.com/paper/2602.14357/) (2026)
- [The Hitchhikers Guide to Production-ready Trustworthy Foundation Model powered Software (FMware)](https://mlsanity.com/paper/2505.10640/) (2025)
- [Software Vulnerability Management in the Era of Artificial Intelligence: An Industry Perspective](https://mlsanity.com/paper/2512.18261/) (2025)
- [LLM-as-an-Annotator: Training Lightweight Models with LLM-Annotated Examples for Aspect Sentiment Tuple Prediction](https://mlsanity.com/paper/2603.01778/) (2026)
- [CHILLGuard: Towards Fine-Grained Chinese LLM Safety Guardrail with Scalable Data Construction and Model-aware Preference Alignment](https://mlsanity.com/paper/2606.15396/) (2026)
- [All Green, Still Broken: Real-Flow Verification Lessons from an LLM-Integrated, Multi-Market Web Application](https://mlsanity.com/paper/2606.22475/) (2026)
- [How Do AI Coding Agents Contribute to Software Development? an Empirical Study of Agentic Pull Requests](https://mlsanity.com/paper/2607.21832/) (2026)
- [AiFlow: Token-Native Reactive Orchestration with Bounded Backpressure for Streaming LLM Applications](https://mlsanity.com/paper/2608.00558/) (2026)
- [Configuring RAG Systems: What Breaks, Where Originates, and How to Fix](https://mlsanity.com/paper/2610.18567/) (2026)

## Abstract (arXiv)

In recent years, large language models (LLMs) have seen rapid advancements, significantly impacting various fields such as computer vision, natural language processing, and software engineering. These LLMs, exemplified by OpenAI's ChatGPT, have revolutionized the way we approach language understanding and generation tasks. However, in contrast to traditional software development practices, LLM development introduces new challenges for AI developers in design, implementation, and deployment. These challenges span different areas (such as prompts, APIs, and plugins), requiring developers to navigate unique methodologies and considerations specific to LLM application development. Despite the profound influence of LLMs, to the best of our knowledge, these challenges have not been thoroughly investigated in previous empirical studies. To fill this gap, we present the first comprehensive study on understanding the challenges faced by LLM developers. Specifically, we crawl and analyze 29,057 relevant questions from a popular OpenAI developer forum. We first examine their popularity and difficulty. After manually analyzing 2,364 sampled questions, we construct a taxonomy of challenges faced by LLM developers. Based on this taxonomy, we summarize a set of findings and actionable implications for LLM-related stakeholders, including developers and providers (especially the OpenAI organization).