mlsanityData to 7 Oct 2026

CodeCoT: Tackling Code Syntax Errors in CoT Reasoning for Code Generation

arXiv 2308.08784 ↗First posted 17 Aug 2023cs.SE cs.AIcited by 29

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Chain-of-thought (CoT) has emerged as a groundbreaking tool in NLP, notably for its efficacy in complex reasoning tasks, such as mathematical proofs. However, its application in code generation faces a distinct challenge, i.e., although the code generated with CoT reasoning is logically correct, it faces the problem of syntax error (e.g., invalid syntax error report) during code execution, which causes the CoT result's pass@1 in HumanEval even lower than the zero-shot result. In this paper, we present Code Chain-of-Thought (CodeCoT) that integrates CoT with a self-examination process for code generation. CodeCoT begins with the LLMs using CoT for initial code development to ensure the generated code follows the correct logic flow. Then, CodeCoT will generate test cases to validate whether the code has syntax errors during the execution. CodeCoT then employs a self-examination phase, in which the generated code is executed against these test cases in the local environment. If the local environment raises error information (e.g., invalid syntax error), CodeCoT will iteratively refine the code based on the feedback information. Within this loop, CodeCoT can make sure their generated codes not only follow the logic flow of the code description, but the syntax error will also be addressed with the self-examination process. Our evaluation results reveal that CodeCoT improves the effectiveness of code generation. For example, CodeCoT increases pass@1 from 75.6% to 79.3% for the HumanEval dataset.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 52 of its 58 references that are in the map. First posted on arXiv 17 Aug 2023.

Builds on

52 of 58 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Reasoning with Language Model is Planning with World Model24 May
  2. 2023Structured Chain-of-Thought Prompting for Code Generation11 May
  3. 2023Teaching Large Language Models to Self-Debug11 Apr
  4. 2023LEVER: Learning to Verify Language-to-Code Generation with Execution16 Feb
  5. 2022Competition-Level Code Generation with AlphaCode8 Feb
  6. 2022Reasoning Like Program Executors27 Jan

Other references · 46

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  9. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  10. 2021Program Synthesis with Large Language Modelscited by 3,509
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  12. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
34 more
  1. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  3. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  4. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  5. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  6. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  7. 2022Large Language Models Can Self-Improvecited by 656
  8. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  9. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  10. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  11. 2022CodeT: Code Generation with Generated Testscited by 369
  12. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  13. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 294
  14. 2022Generating Sequences by Learning to Self-Correctcited by 271
  15. 2023Self-collaboration Code Generation via ChatGPTcited by 255
  16. 2023Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender Systemcited by 241
  17. 2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 231
  18. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 202
  19. 2023Is Self-Repair a Silver Bullet for Code Generation?cited by 191
  20. 2023Active Prompting with Chain-of-Thought for Large Language Modelscited by 154
  21. 2022Natural Language to Code Translation with Executioncited by 122
  22. 2023Self-planning Code Generation with Large Language Modelscited by 121
  23. 2023Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Datacited by 119
  24. 2023Self-Edit: Fault-Aware Code Editor for Code Generationcited by 110
  25. 2022Coder Reviewer Reranking for Code Generationcited by 83
  26. 2023Improving Code Generation by Training with Natural Language Feedbackcited by 75
  27. 2023CodeScore: Evaluating Code Generation by Learning Code Executioncited by 66
  28. 2023Eliciting the Translation Ability of Large Language Models via Multilingual Finetuning with Translation Instructionscited by 55
  29. 2021MSP: Multi-Stage Prompting for Making Pre-trained Language Models Better Translatorscited by 35
  30. 2022Fault-Aware Neural Code Rankerscited by 33
  31. 2021GPT3-to-plan: Extracting plans from text using GPT-3cited by 26
  32. 2022Multi-Stage Prompting for Knowledgeable Dialogue Generationcited by 22
  33. 2020Stolen Probability: A Structural Weakness of Neural Language Modelscited by 21
  34. 2023Efficient Prompting via Dynamic In-Context Learningcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 17; 2026: 5.

29 papers.

Sort:

By area:

  1. 2025VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Modelscited by 102
  2. 2024Agents in Software Engineering: Survey, Landscape, and Visioncited by 60
  3. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47
  4. 2024A Pair Programming Framework for Code Generation via Multi-Plan Exploration and Feedback-Driven Refinementcited by 21
  5. 2025CoCoEvo: Co-Evolution of Programs and Test Cases to Enhance Code Generationcited by 14
  6. 2024RGD: Multi-LLM Based Agent Debugger via Refinement and Generation Guidancecited by 12
  7. 2025Reasoning-as-Logic-Units: Scaling Test-Time Reasoning in Large Language Models Through Logic Unit Alignmentcited by 12
  8. 2025Rethinking Verification for LLM Code Generation: From Generation to Testingcited by 12
  9. 2025Large Language Model Guided Self-Debugging Code Generationcited by 7
  10. 2025HoarePrompt: Structural Reasoning About Program Correctness in Natural Languagecited by 7
  11. 2026LLM-Based Agentic Systems for Software Engineering: Challenges and Opportunitiescited by 7
  12. 2025Demystifying Errors in LLM Reasoning Traces: An Empirical Study of Code Execution Simulationcited by 5
17 more
  1. 2024A Study on Training and Developing Large Language Models for Behavior Tree Generationcited by 4
  2. 2025Code Reasoning for Software Engineering Tasks: A Survey and A Call to Actioncited by 4
  3. 2024Seed-CTS: Unleashing the Power of Tree Search for Superior Performance in Competitive Coding Taskscited by 3
  4. 2024Thinking Before Running! Efficient Code Generation with Thorough Exploration and Optimal Refinementcited by 3
  5. 2025LexChain: Modeling Legal Reasoning Chains for Chinese Tort Case Analysiscited by 3
  6. 2024Generative transformations and patterns in LLM-native approaches for software verification and falsificationcited by 2
  7. 2025MoT: Modularization-of-Thought Prompting for Effective Code Generationcited by 2
  8. 2025SRLCG: Self-Rectified Large-Scale Code Generation with Multidimensional Chain-of-Thought and Dynamic Backtrackingcited by 2
  9. 2025Intention Chain-of-Thought Prompting with Dynamic Routing for Code Generationcited by 2
  10. 2025Human-Aligned Bench: Fine-Grained Assessment of Reasoning Ability in MLLMs vs. Humanscited by 1
  11. 2025From Context to Intent: Reasoning-Guided Function-Level Code Completioncited by 1
  12. 2025Beyond Language Barriers: Multi-Agent Coordination for Multi-Language Code Generationcited by 1
  13. 2025Reinforcement Learning-Guided Chain-of-Draft for Token-Efficient Code Generationcited by 1
  14. 2026InfEngine: A Self-Verifying and Self-Optimizing Intelligent Engine for Infrared Radiation Computingcited by 0
  15. 2026DuET: Dual Execution for Test Output Prediction with Generated Code and Pseudocodecited by 0
  16. 2026StoryCoder: Narrative Reformulation for Structured Reasoning in LLM Code Generationcited by 0
  17. 2026You Don't Need Public Tests to Generate Correct Codecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Scaling Code-Assisted Chain-of-Thoughts and Instructions for Model Reasoning
  2. 2025Understanding Chain-of-Thought Effectiveness in Code Generation: An Empirical and Information-Theoretic Analysis
  3. 2023Chain-of-Thought in Neural Code Generation: From and For Lightweight Language Models
  4. 2025Generating Verifiable Chain of Thoughts from Exection-Traces
  5. 2025Code Execution as Grounded Supervision for LLM Reasoning
  6. 2025Uncertainty-Guided Chain-of-Thought for Code Generation with LLMs
  7. 2024Faithful Logical Reasoning via Symbolic Chain-of-Thought
  8. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning