mlsanityData to 14 Oct 2026

Is Stack Overflow Obsolete? An Empirical Study of the Characteristics of ChatGPT Answers to Stack Overflow Questions

arXiv 2308.02312 ↗First posted 4 Aug 2023cs.SE cs.AIcited by 42

Not on the map. Why?

Abstract (arXiv)

Q&A platforms have been crucial for the online help-seeking behavior of programmers. However, the recent popularity of ChatGPT is altering this trend. Despite this popularity, no comprehensive study has been conducted to evaluate the characteristics of ChatGPT's answers to programming questions. To bridge the gap, we conducted the first in-depth analysis of ChatGPT answers to 517 programming questions on Stack Overflow and examined the correctness, consistency, comprehensiveness, and conciseness of ChatGPT answers. Furthermore, we conducted a large-scale linguistic analysis, as well as a user study, to understand the characteristics of ChatGPT answers from linguistic and human aspects. Our analysis shows that 52% of ChatGPT answers contain incorrect information and 77% are verbose. Nonetheless, our user study participants still preferred ChatGPT answers 35% of the time due to their comprehensiveness and well-articulated language style. However, they also overlooked the misinformation in the ChatGPT answers 39% of the time. This implies the need to counter misinformation in ChatGPT answers to programming questions and raise awareness of the risks associated with seemingly correct answers.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Builds on 26 of its 87 references that are in the map. First posted on arXiv 4 Aug 2023.

Builds on

26 of 87 references.

References · 26

  1. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  2. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  3. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  4. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  5. 2023The Internal State of an LLM Knows When It's Lyingcited by 659
  6. 2023How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detectioncited by 481
  7. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 364
  8. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 353
  9. 2017DeepPath: A Reinforcement Learning Method for Knowledge Graph Reasoningcited by 266
  10. 2022Grounded Copilot: How Programmers Interact with Code-Generating Modelscited by 218
  11. 2023ChatGPT: Jack of all trades, master of nonecited by 204
  12. 2023A Categorical Archive of ChatGPT Failurescited by 168
14 more
  1. 2022GitHub Copilot AI pair programmer: Asset or Liability?cited by 152
  2. 2021Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Basescited by 107
  3. 2019Assessing The Factual Accuracy of Generated Textcited by 103
  4. 2023The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Developmentcited by 96
  5. 2018'It's Reducing a Human Being to a Percentage'; Perceptions of Justice in Algorithmic Decisionscited by 91
  6. 2023ChatGPT or Human? Detect and Explain. Explaining Decisions of Machine Learning Model for Detecting Short ChatGPT-generated Textcited by 85
  7. 2021The Factual Inconsistency Problem in Abstractive Text Summarization: A Surveycited by 79
  8. 2022Are Deepfakes Concerning? Analyzing Conversations of Deepfakes on Reddit and Exploring Societal Implicationscited by 14
  9. 2017Sentiment Polarity Detection for Software Developmentcited by 12
  10. 2023Human-Centered Responsible Artificial Intelligence: Current & Future Trendscited by 8
  11. 2018Evaluating How Developers Use General-Purpose Web-Search for Code Retrievalcited by 6
  12. 2023Automated Summarization of Stack Overflow Postscited by 5
  13. 2017How to Ask for Technical Help? Evidence-based Guidelines for Writing Questions on Stack Overflowcited by 4
  14. 2020Quality Estimation & Interpretability for Code Translationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 8; 2025: 16; 2026: 13.

42 papers.

Sort:

By area:

  1. 2023Human Feedback is not Gold Standardcited by 83
  2. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 71
  3. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  4. 2023AI-TA: Towards an Intelligent Question-Answer Teaching Assistant using Open-Source LLMscited by 22
  5. 2025Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Reviewcited by 16
  6. 2024SM3-Text-to-Query: Synthetic Multi-Model Medical Text-to-Query Benchmarkcited by 14
  7. 2025ComPO: Preference Alignment via Comparison Oraclescited by 13
  8. 2025How Do Programming Students Use Generative AI?cited by 10
  9. 2025Do LLMs Consider Security? An Empirical Study on Responses to Programming Questionscited by 9
  10. 2025Rubrik's Cube: Testing a New Rubric for Evaluating Explanations on the CUBE datasetcited by 8
  11. 2024Synthetic Human Memories: AI-Edited Images and Videos Can Implant False Memories and Distort Recollectioncited by 7
  12. 2025How Individual Traits and Language Styles Shape Preferences In Open-ended User-LLM Interaction: A Preliminary Studycited by 7
30 more
  1. 2025Linguistic Characteristics of AI-Generated Text: A Surveycited by 7
  2. 2023GenCodeSearchNet: A Benchmark Test Suite for Evaluating Generalization in Programming Language Understandingcited by 6
  3. 2024An Insight into Security Code Review with LLMs: Capabilities, Obstacles, and Influential Factorscited by 6
  4. 2025How Safe Are AI-Generated Patches? A Large-scale Study on Security Risks in LLM and Agentic Automated Program Repair on SWE-benchcited by 5
  5. 2025Collaborative Document Editing with Multiple Users and AI Agentscited by 5
  6. 2024ChatGPT's Potential in Cryptography Misuse Detection: A Comparative Analysis with Static Analysis Toolscited by 4
  7. 2025Using LLMs in Generating Design Rationale for Software Architecture Decisionscited by 4
  8. 2025Curiosity by Design: An LLM-based Coding Assistant Asking Clarification Questionscited by 4
  9. 2023Complementary Advantages of ChatGPTs and Human Readers in Reasoning: Evidence from English Text Reading Comprehensioncited by 2
  10. 2024The Heterogeneous Productivity Effects of Generative AIcited by 2
  11. 2025Information Retrieval in the Age of Generative AI: The RGB Modelcited by 2
  12. 2025Walking the Tightrope of LLMs for Software Development: A Practitioners' Perspectivecited by 2
  13. 2023Transforming the Output of Generative Pre-trained Transformer: The Influence of the PGI Framework on Attention Dynamicscited by 1
  14. 2024A Risk Taxonomy and Reflection Tool for Large Language Model Adoption in Public Healthcited by 1
  15. 2025Let Them Down Easy! Contextual Effects of LLM Guardrails on User Perceptions and Preferencescited by 1
  16. 2025User Misconceptions of LLM-Based Conversational Programming Assistantscited by 1
  17. 2026Illocutionary Explanation Planning for Source-Faithful Explanations in Retrieval-Augmented Language Modelscited by 1
  18. 2026Beyond the 'Diff': Addressing Agentic Entropy in Agentic Software Developmentcited by 1
  19. 2025Cracking CodeWhisperer: Analyzing Developers' Interactions and Patterns During Programming Taskscited by 0
  20. 2026Human-Aligned Enhancement of Programming Answers with LLMs Guided by User Feedbackcited by 0
  21. 2026Tricky2: Towards a Benchmark for Evaluating Human and LLM Error Interactionscited by 0
  22. 2026Inference-Time Safety For Code LLMs Via Retrieval-Augmented Revisioncited by 0
  23. 2026Evaluating LLMs' Effectiveness on Real-World Consumer Device Repair Questionscited by 0
  24. 2026LLMoxie: Exploring Agentic AI for Scientific Software Developmentcited by 0
  25. 2026Aleena: Alignment Agent for Research Software Engineering Collaborationscited by 0
  26. 2026When LLMs Over-Answer: Measuring and Mitigating Quality Issues in LLM-Based Hardware Description Language Question Answeringcited by 0
  27. 2026AI Slop and Hallucinations in Vulnerability Assessment: A Survey on Reasoning Failures and Trustworthy Mitigationcited by 0
  28. 2026CoSTAR: Data Synthesis-Driven Constraint-Aware COBOL Section Summarization for Legacy System Modernizationcited by 0
  29. 2026A Zeroth-Order Paradigm for LLM Preference Alignmentcited by 0
  30. 2026Complexity-Aware Evaluation of LLM Comprehensioncited by 0