# Is Stack Overflow Obsolete? An Empirical Study of the Characteristics of ChatGPT Answers to Stack Overflow Questions

arXiv: https://arxiv.org/abs/2308.02312 · HTML: https://mlsanity.com/paper/2308.02312/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 87 references that are in the map.
- First posted on arXiv 4 Aug 2023.

## Builds on (most cited first, 26 of 26)

- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [The Internal State of an LLM Knows When It's Lying](https://mlsanity.com/paper/2304.13734/) (2023)
- [How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection](https://mlsanity.com/paper/2301.07597/) (2023)
- [Measuring and Improving Consistency in Pretrained Language Models](https://mlsanity.com/paper/2102.01017/) (2021)
- [Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback](https://mlsanity.com/paper/2302.12813/) (2023)
- [DeepPath: A Reinforcement Learning Method for Knowledge Graph Reasoning](https://mlsanity.com/paper/1707.06690/) (2017)
- [Grounded Copilot: How Programmers Interact with Code-Generating Models](https://mlsanity.com/paper/2206.15000/) (2022)
- [ChatGPT: Jack of all trades, master of none](https://mlsanity.com/paper/2302.10724/) (2023)
- [A Categorical Archive of ChatGPT Failures](https://mlsanity.com/paper/2302.03494/) (2023)
- [GitHub Copilot AI pair programmer: Asset or Liability?](https://mlsanity.com/paper/2206.15331/) (2022)
- [Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases](https://mlsanity.com/paper/2106.09231/) (2021)
- [Assessing The Factual Accuracy of Generated Text](https://mlsanity.com/paper/1905.13322/) (2019)
- [The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Development](https://mlsanity.com/paper/2302.07080/) (2023)
- ['It's Reducing a Human Being to a Percentage'; Perceptions of Justice in Algorithmic Decisions](https://mlsanity.com/paper/1801.10408/) (2018)
- [ChatGPT or Human? Detect and Explain. Explaining Decisions of Machine Learning Model for Detecting Short ChatGPT-generated Text](https://mlsanity.com/paper/2301.13852/) (2023)
- [The Factual Inconsistency Problem in Abstractive Text Summarization: A Survey](https://mlsanity.com/paper/2104.14839/) (2021)
- [Are Deepfakes Concerning? Analyzing Conversations of Deepfakes on Reddit and Exploring Societal Implications](https://mlsanity.com/paper/2203.15044/) (2022)
- [Sentiment Polarity Detection for Software Development](https://mlsanity.com/paper/1709.02984/) (2017)
- [Human-Centered Responsible Artificial Intelligence: Current & Future Trends](https://mlsanity.com/paper/2302.08157/) (2023)
- [Evaluating How Developers Use General-Purpose Web-Search for Code Retrieval](https://mlsanity.com/paper/1803.08612/) (2018)
- [Automated Summarization of Stack Overflow Posts](https://mlsanity.com/paper/2305.16680/) (2023)
- [How to Ask for Technical Help? Evidence-based Guidelines for Writing Questions on Stack Overflow](https://mlsanity.com/paper/1710.04692/) (2017)
- [Quality Estimation & Interpretability for Code Translation](https://mlsanity.com/paper/2012.07581/) (2020)

## Built on by (most cited first, 30 of 42)

- [Human Feedback is not Gold Standard](https://mlsanity.com/paper/2309.16349/) (2023)
- [When LLMs Meet Cybersecurity: A Systematic Literature Review](https://mlsanity.com/paper/2405.03644/) (2024)
- [A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications](https://mlsanity.com/paper/2410.15595/) (2024)
- [AI-TA: Towards an Intelligent Question-Answer Teaching Assistant using Open-Source LLMs](https://mlsanity.com/paper/2311.02775/) (2023)
- [Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Review](https://mlsanity.com/paper/2501.12557/) (2025)
- [SM3-Text-to-Query: Synthetic Multi-Model Medical Text-to-Query Benchmark](https://mlsanity.com/paper/2411.05521/) (2024)
- [ComPO: Preference Alignment via Comparison Oracles](https://mlsanity.com/paper/2505.05465/) (2025)
- [How Do Programming Students Use Generative AI?](https://mlsanity.com/paper/2501.10091/) (2025)
- [Do LLMs Consider Security? An Empirical Study on Responses to Programming Questions](https://mlsanity.com/paper/2502.14202/) (2025)
- [Rubrik's Cube: Testing a New Rubric for Evaluating Explanations on the CUBE dataset](https://mlsanity.com/paper/2503.23899/) (2025)
- [Synthetic Human Memories: AI-Edited Images and Videos Can Implant False Memories and Distort Recollection](https://mlsanity.com/paper/2409.08895/) (2024)
- [How Individual Traits and Language Styles Shape Preferences In Open-ended User-LLM Interaction: A Preliminary Study](https://mlsanity.com/paper/2504.17083/) (2025)
- [Linguistic Characteristics of AI-Generated Text: A Survey](https://mlsanity.com/paper/2510.05136/) (2025)
- [GenCodeSearchNet: A Benchmark Test Suite for Evaluating Generalization in Programming Language Understanding](https://mlsanity.com/paper/2311.09707/) (2023)
- [An Insight into Security Code Review with LLMs: Capabilities, Obstacles, and Influential Factors](https://mlsanity.com/paper/2401.16310/) (2024)
- [How Safe Are AI-Generated Patches? A Large-scale Study on Security Risks in LLM and Agentic Automated Program Repair on SWE-bench](https://mlsanity.com/paper/2507.02976/) (2025)
- [Collaborative Document Editing with Multiple Users and AI Agents](https://mlsanity.com/paper/2509.11826/) (2025)
- [ChatGPT's Potential in Cryptography Misuse Detection: A Comparative Analysis with Static Analysis Tools](https://mlsanity.com/paper/2409.06561/) (2024)
- [Using LLMs in Generating Design Rationale for Software Architecture Decisions](https://mlsanity.com/paper/2504.20781/) (2025)
- [Curiosity by Design: An LLM-based Coding Assistant Asking Clarification Questions](https://mlsanity.com/paper/2507.21285/) (2025)
- [Complementary Advantages of ChatGPTs and Human Readers in Reasoning: Evidence from English Text Reading Comprehension](https://mlsanity.com/paper/2311.10344/) (2023)
- [The Heterogeneous Productivity Effects of Generative AI](https://mlsanity.com/paper/2403.01964/) (2024)
- [Information Retrieval in the Age of Generative AI: The RGB Model](https://mlsanity.com/paper/2504.20610/) (2025)
- [Walking the Tightrope of LLMs for Software Development: A Practitioners' Perspective](https://mlsanity.com/paper/2511.06428/) (2025)
- [Transforming the Output of Generative Pre-trained Transformer: The Influence of the PGI Framework on Attention Dynamics](https://mlsanity.com/paper/2308.13317/) (2023)
- [A Risk Taxonomy and Reflection Tool for Large Language Model Adoption in Public Health](https://mlsanity.com/paper/2411.02594/) (2024)
- [Let Them Down Easy! Contextual Effects of LLM Guardrails on User Perceptions and Preferences](https://mlsanity.com/paper/2506.00195/) (2025)
- [User Misconceptions of LLM-Based Conversational Programming Assistants](https://mlsanity.com/paper/2510.25662/) (2025)
- [Illocutionary Explanation Planning for Source-Faithful Explanations in Retrieval-Augmented Language Models](https://mlsanity.com/paper/2604.06211/) (2026)
- [Beyond the 'Diff': Addressing Agentic Entropy in Agentic Software Development](https://mlsanity.com/paper/2604.16323/) (2026)

## Abstract (arXiv)

Q&A platforms have been crucial for the online help-seeking behavior of programmers. However, the recent popularity of ChatGPT is altering this trend. Despite this popularity, no comprehensive study has been conducted to evaluate the characteristics of ChatGPT's answers to programming questions. To bridge the gap, we conducted the first in-depth analysis of ChatGPT answers to 517 programming questions on Stack Overflow and examined the correctness, consistency, comprehensiveness, and conciseness of ChatGPT answers. Furthermore, we conducted a large-scale linguistic analysis, as well as a user study, to understand the characteristics of ChatGPT answers from linguistic and human aspects. Our analysis shows that 52% of ChatGPT answers contain incorrect information and 77% are verbose. Nonetheless, our user study participants still preferred ChatGPT answers 35% of the time due to their comprehensiveness and well-articulated language style. However, they also overlooked the misinformation in the ChatGPT answers 39% of the time. This implies the need to counter misinformation in ChatGPT answers to programming questions and raise awareness of the risks associated with seemingly correct answers.