mlsanityData to 14 Oct 2026

An Empirical Study of the Non-determinism of ChatGPT in Code Generation

arXiv 2308.02828 ↗First posted 5 Aug 2023cs.SEcited by 137

Not on the map. Why?

Cited by 24 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

There has been a recent explosion of research on Large Language Models (LLMs) for software engineering tasks, in particular code generation. However, results from LLMs can be highly unstable; nondeterministically returning very different codes for the same prompt. Non-determinism is a potential menace to scientific conclusion validity. When non-determinism is high, scientific conclusions simply cannot be relied upon unless researchers change their behaviour to control for it in their empirical analyses. This paper conducts an empirical study to demonstrate that non-determinism is, indeed, high, thereby underlining the need for this behavioural change. We choose to study ChatGPT because it is already highly prevalent in the code generation research literature. We report results from a study of 829 code generation problems from three code generation benchmarks (i.e., CodeContests, APPS, and HumanEval). Our results reveal high degrees of non-determinism: the ratio of coding tasks with zero equal test output across different requests is 75.76%, 51.00%, and 47.56% for CodeContests, APPS, and HumanEval, respectively. In addition, we find that setting the temperature to 0 does not guarantee determinism in code generation, although it indeed brings less non-determinism than the default configuration (temperature=1). These results confirm that there is, currently, a significant threat to scientific conclusion validity. In order to put LLM-based research on firmer scientific foundations, researchers need to take into account non-determinism in drawing their conclusions.

Key facts. Cited by 137 papers in the map (data through 14 Oct 2026). First posted on arXiv 5 Aug 2023.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 32; 2025: 46; 2026: 50.

137 papers.

Sort:

By area:

  1. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 399
  2. 2024LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contextscited by 245
  3. 2024Non-Determinism of "Deterministic" LLM Settingscited by 92
  4. 2023Are You Sure? Challenging LLMs Leads to Performance Drops in The FlipFlop Experimentcited by 77
  5. 2024CogBench: a large language model walks into a psychology labcited by 63
  6. 2024Query-Based Adversarial Prompt Generationcited by 62
  7. 2025Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inferencecited by 53
  8. 2024EDT: Improving Large Language Models' Generation by Entropy-based Dynamic Temperature Samplingcited by 45
  9. 2023Exploring the Sensitivity of LLMs' Decision-Making Capabilities: Insights from Prompt Variation and Hyperparameterscited by 32
  10. 2024CodeMind: Evaluating Large Language Models for Code Reasoningcited by 32
  11. 2024Easy Problems That LLMs Get Wrongcited by 30
  12. 2025Large Language Models Miss the Multi-Agent Markcited by 26
38 more
  1. 2024What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Usecited by 24
  2. 2025Hallucination Detection in Large Language Models with Metamorphic Relationscited by 24
  3. 2023(Why) Is My Prompt Getting Worse? Rethinking Regression Testing for Evolving LLM APIscited by 23
  4. 2024CodeMirage: Hallucinations in Code Generated by Large Language Modelscited by 23
  5. 2025AgentBound: Securing Execution Boundaries of AI Agentscited by 23
  6. 2025QualityFlow: An Agentic Workflow for Program Synthesis Controlled by LLM Quality Checkscited by 22
  7. 2023METAL: Metamorphic Testing Framework for Analyzing Large-Language Model Qualitiescited by 20
  8. 2025Don't Judge Code by Its Cover: Exploring Biases in LLM Judges for Code Evaluationcited by 18
  9. 2024Scaling Up Video Summarization Pretraining with Large Language Modelscited by 17
  10. 2026On Randomness in Agentic Evalscited by 17
  11. 2025Give LLMs a Security Course: Securing Retrieval-Augmented Code Generation via Knowledge Injectioncited by 16
  12. 2025Exploring the Potential of Diffusion Large Language Models in Code Generationcited by 16
  13. 2025An Empirical Study on Failures in Automated Issue Solvingcited by 16
  14. 2023Benchmarking and Explaining Large Language Model-based Code Generation: A Causality-Centric Approachcited by 15
  15. 2024HLM-Cite: Hybrid Language Model Workflow for Text-based Scientific Citation Predictioncited by 15
  16. 2024MAG-V: A Multi-Agent Framework for Synthetic Data Generation and Verificationcited by 12
  17. 2025Reflections on the Reproducibility of Commercial LLM Performance in Empirical Software Engineering Studiescited by 12
  18. 2024Fairness Testing of Large Language Models in Role-Playingcited by 11
  19. 2025EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devicescited by 11
  20. 2026TrajAudit: Automated Failure Diagnosis for Agentic Coding Systemscited by 11
  21. 2023Language Models as a Service: Overview of a New Paradigm and its Challengescited by 10
  22. 2025Agentic AI Process Observability: Discovering Behavioral Variabilitycited by 10
  23. 2025Failure Modes in LLM Systems: A System-Level Taxonomy for Reliable AI Applicationscited by 10
  24. 2024Guiding Enumerative Program Synthesis with Large Language Modelscited by 9
  25. 2024Statically Contextualizing Large Language Models with Typed Holescited by 9
  26. 2025Exploring the Security Threats of Knowledge Base Poisoning in Retrieval-Augmented Code Generationcited by 9
  27. 2026Replayable Financial Agents: A Determinism-Faithfulness Assurance Harness for Tool-Using LLM Agentscited by 7
  28. 2026Toward Expert Investment Teams:A Multi-Agent LLM System with Fine-Grained Trading Taskscited by 7
  29. 2024An Insight into Security Code Review with LLMs: Capabilities, Obstacles, and Influential Factorscited by 6
  30. 2024Rethinking Software Engineering in the Foundation Model Era: A Curated Catalogue of Challenges in the Development of Trustworthy FMwarecited by 6
  31. 2024Validating LLM-Generated Programs with Metamorphic Prompt Testingcited by 6
  32. 2024The Cognitive Capabilities of Generative AI: A Comparative Analysis with Human Benchmarkscited by 6
  33. 2026When Agents Fail: A Comprehensive Study of Bugs in LLM Agents with Automated Labelingcited by 6
  34. 2024Search-based Optimisation of LLM Learning Shots for Story Point Estimationcited by 5
  35. 2025Towards Advancing Code Generation with Large Language Models: A Research Roadmapcited by 5
  36. 2025AutoMisty: A Multi-Agent LLM Framework for Automated Code Generation in the Misty Social Robotcited by 5
  37. 2024Generative AI to Generate Test Data Generatorscited by 4
  38. 2024To what extent is ChatGPT useful for language teacher lesson plan creation?cited by 4

50 most cited of 137 All 137 (JSON)