# An Empirical Study of OpenAI API Discussions on Stack Overflow

arXiv: https://arxiv.org/abs/2505.04084 · HTML: https://mlsanity.com/paper/2505.04084/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 50 references that are in the map.
- First posted on arXiv 7 May 2025.

## Builds on (most cited first, 15 of 15)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [Large Language Models for Software Engineering: Survey and Open Problems](https://mlsanity.com/paper/2310.03533/) (2023)
- [Software Testing with Large Language Models: Survey, Landscape, and Vision](https://mlsanity.com/paper/2307.07221/) (2023)
- [A Survey on Large Language Models for Software Engineering](https://mlsanity.com/paper/2312.15223/) (2023)
- [The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Survey](https://mlsanity.com/paper/2401.07872/) (2024)
- [LLMs Meet Library Evolution: Evaluating Deprecated API Usage in LLM-based Code Completion](https://mlsanity.com/paper/2406.09834/) (2024)
- [Are Large Language Models a Threat to Digital Public Goods? Evidence from Activity on Stack Overflow](https://mlsanity.com/paper/2307.07367/) (2023)
- [An Empirical Study on Challenges for LLM Application Developers](https://mlsanity.com/paper/2408.05002/) (2024)
- [What Do Developers Ask About ML Libraries? A Large-scale Study Using Stack Overflow](https://mlsanity.com/paper/1906.11940/) (2019)
- [Towards Enhancing the Reproducibility of Deep Learning Bugs: An Empirical Study](https://mlsanity.com/paper/2401.03069/) (2024)
- [Common Challenges of Deep Reinforcement Learning Applications Development: An Empirical Study](https://mlsanity.com/paper/2310.09575/) (2023)
- [Unveiling the Impact of Multi-Modal Interactions on User Engagement: A Comprehensive Evaluation in AI-driven Conversations](https://mlsanity.com/paper/2406.15000/) (2024)
- [Challenges in Docker Development: A Large-scale Study Using Stack Overflow](https://mlsanity.com/paper/2008.04467/) (2020)
- [Developer Challenges on Large Language Models: A Study of Stack Overflow and OpenAI Developer Forum Posts](https://mlsanity.com/paper/2411.10873/) (2024)

## Built on by (most cited first, 1 of 1)

- [Why Does the LLM Stop Computing: An Empirical Study of User-Reported Failures in Open-Source LLMs](https://mlsanity.com/paper/2601.13655/) (2026)

## Abstract (arXiv)

The rapid advancement of large language models (LLMs), represented by OpenAI's GPT series, has significantly impacted various domains such as natural language processing, software development, education, healthcare, finance, and scientific research. However, OpenAI APIs introduce unique challenges that differ from traditional APIs, such as the complexities of prompt engineering, token-based cost management, non-deterministic outputs, and operation as black boxes. To the best of our knowledge, the challenges developers encounter when using OpenAI APIs have not been explored in previous empirical studies. To fill this gap, we conduct the first comprehensive empirical study by analyzing 2,874 OpenAI API-related discussions from the popular Q&A forum Stack Overflow. We first examine the popularity and difficulty of these posts. After manually categorizing them into nine OpenAI API-related categories, we identify specific challenges associated with each category through topic modeling analysis. Based on our empirical findings, we finally propose actionable implications for developers, LLM vendors, and researchers.