An Empirical Study of OpenAI API Discussions on Stack Overflow
Not on the map. Why?
Abstract (arXiv)
The rapid advancement of large language models (LLMs), represented by OpenAI's GPT series, has significantly impacted various domains such as natural language processing, software development, education, healthcare, finance, and scientific research. However, OpenAI APIs introduce unique challenges that differ from traditional APIs, such as the complexities of prompt engineering, token-based cost management, non-deterministic outputs, and operation as black boxes. To the best of our knowledge, the challenges developers encounter when using OpenAI APIs have not been explored in previous empirical studies. To fill this gap, we conduct the first comprehensive empirical study by analyzing 2,874 OpenAI API-related discussions from the popular Q&A forum Stack Overflow. We first examine the popularity and difficulty of these posts. After manually categorizing them into nine OpenAI API-related categories, we identify specific challenges associated with each category through topic modeling analysis. Based on our empirical findings, we finally propose actionable implications for developers, LLM vendors, and researchers.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 15 of its 50 references that are in the map. First posted on arXiv 7 May 2025.
Builds on
15 of 50 references.
References · 15
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
- 2023Large Language Models for Software Engineering: Survey and Open Problemscited by 178
- 2023Software Testing with Large Language Models: Survey, Landscape, and Visioncited by 163
- 2023A Survey on Large Language Models for Software Engineeringcited by 58
- 2024The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Surveycited by 32
- 2024LLMs Meet Library Evolution: Evaluating Deprecated API Usage in LLM-based Code Completioncited by 29
- 2023Are Large Language Models a Threat to Digital Public Goods? Evidence from Activity on Stack Overflowcited by 22
- 2024An Empirical Study on Challenges for LLM Application Developerscited by 17
- 2019What Do Developers Ask About ML Libraries? A Large-scale Study Using Stack Overflowcited by 13
- 2024Towards Enhancing the Reproducibility of Deep Learning Bugs: An Empirical Studycited by 6
- 2023Common Challenges of Deep Reinforcement Learning Applications Development: An Empirical Studycited by 5
3 more
- 2024Unveiling the Impact of Multi-Modal Interactions on User Engagement: A Comprehensive Evaluation in AI-driven Conversationscited by 4
- 2020Challenges in Docker Development: A Large-scale Study Using Stack Overflowcited by 3
- 2024Developer Challenges on Large Language Models: A Study of Stack Overflow and OpenAI Developer Forum Postscited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.