# CodeCoT: Tackling Code Syntax Errors in CoT Reasoning for Code Generation

arXiv: https://arxiv.org/abs/2308.08784 · HTML: https://mlsanity.com/paper/2308.08784/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 52 of its 58 references that are in the map.
- First posted on arXiv 17 Aug 2023.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 30 of 52)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://mlsanity.com/paper/2112.06905/) (2021)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [Reasoning with Language Model is Planning with World Model](https://mlsanity.com/paper/2305.14992/) (2023)
- [Automatic Chain of Thought Prompting in Large Language Models](https://mlsanity.com/paper/2210.03493/) (2022)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017)
- [Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](https://mlsanity.com/paper/2201.11990/) (2022)
- [InCoder: A Generative Model for Code Infilling and Synthesis](https://mlsanity.com/paper/2204.05999/) (2022)
- [CodeT: Code Generation with Generated Tests](https://mlsanity.com/paper/2207.10397/) (2022)
- [CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning](https://mlsanity.com/paper/2207.01780/) (2022)
- [CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X](https://mlsanity.com/paper/2303.17568/) (2023)
- [Generating Sequences by Learning to Self-Correct](https://mlsanity.com/paper/2211.00053/) (2022)
- [Self-collaboration Code Generation via ChatGPT](https://mlsanity.com/paper/2304.07590/) (2023)

## Built on by (most cited first, 29 of 29)

- [VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models](https://mlsanity.com/paper/2504.15279/) (2025)
- [Agents in Software Engineering: Survey, Landscape, and Vision](https://mlsanity.com/paper/2409.09030/) (2024)
- [Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMs](https://mlsanity.com/paper/2502.19411/) (2025)
- [A Pair Programming Framework for Code Generation via Multi-Plan Exploration and Feedback-Driven Refinement](https://mlsanity.com/paper/2409.05001/) (2024)
- [CoCoEvo: Co-Evolution of Programs and Test Cases to Enhance Code Generation](https://mlsanity.com/paper/2502.10802/) (2025)
- [Rethinking Verification for LLM Code Generation: From Generation to Testing](https://mlsanity.com/paper/2507.06920/) (2025)
- [RGD: Multi-LLM Based Agent Debugger via Refinement and Generation Guidance](https://mlsanity.com/paper/2410.01242/) (2024)
- [Reasoning-as-Logic-Units: Scaling Test-Time Reasoning in Large Language Models Through Logic Unit Alignment](https://mlsanity.com/paper/2502.07803/) (2025)
- [Large Language Model Guided Self-Debugging Code Generation](https://mlsanity.com/paper/2502.02928/) (2025)
- [LLM-Based Agentic Systems for Software Engineering: Challenges and Opportunities](https://mlsanity.com/paper/2601.09822/) (2026)
- [HoarePrompt: Structural Reasoning About Program Correctness in Natural Language](https://mlsanity.com/paper/2503.19599/) (2025)
- [Demystifying Errors in LLM Reasoning Traces: An Empirical Study of Code Execution Simulation](https://mlsanity.com/paper/2512.00215/) (2025)
- [A Study on Training and Developing Large Language Models for Behavior Tree Generation](https://mlsanity.com/paper/2401.08089/) (2024)
- [Code Reasoning for Software Engineering Tasks: A Survey and A Call to Action](https://mlsanity.com/paper/2506.13932/) (2025)
- [Seed-CTS: Unleashing the Power of Tree Search for Superior Performance in Competitive Coding Tasks](https://mlsanity.com/paper/2412.12544/) (2024)
- [Thinking Before Running! Efficient Code Generation with Thorough Exploration and Optimal Refinement](https://mlsanity.com/paper/2502.17442/) (2024)
- [LexChain: Modeling Legal Reasoning Chains for Chinese Tort Case Analysis](https://mlsanity.com/paper/2510.17602/) (2025)
- [Generative transformations and patterns in LLM-native approaches for software verification and falsification](https://mlsanity.com/paper/2404.09384/) (2024)
- [MoT: Modularization-of-Thought Prompting for Effective Code Generation](https://mlsanity.com/paper/2503.12483/) (2025)
- [SRLCG: Self-Rectified Large-Scale Code Generation with Multidimensional Chain-of-Thought and Dynamic Backtracking](https://mlsanity.com/paper/2504.00532/) (2025)
- [Beyond Language Barriers: Multi-Agent Coordination for Multi-Language Code Generation](https://mlsanity.com/paper/2509.19918/) (2025)
- [Intention Chain-of-Thought Prompting with Dynamic Routing for Code Generation](https://mlsanity.com/paper/2512.14048/) (2025)
- [Human-Aligned Bench: Fine-Grained Assessment of Reasoning Ability in MLLMs vs. Humans](https://mlsanity.com/paper/2505.11141/) (2025)
- [From Context to Intent: Reasoning-Guided Function-Level Code Completion](https://mlsanity.com/paper/2508.09537/) (2025)
- [Reinforcement Learning-Guided Chain-of-Draft for Token-Efficient Code Generation](https://mlsanity.com/paper/2509.25243/) (2025)
- [InfEngine: A Self-Verifying and Self-Optimizing Intelligent Engine for Infrared Radiation Computing](https://mlsanity.com/paper/2602.18985/) (2026)
- [DuET: Dual Execution for Test Output Prediction with Generated Code and Pseudocode](https://mlsanity.com/paper/2604.11514/) (2026)
- [StoryCoder: Narrative Reformulation for Structured Reasoning in LLM Code Generation](https://mlsanity.com/paper/2604.14631/) (2026)
- [You Don't Need Public Tests to Generate Correct Code](https://mlsanity.com/paper/2604.21598/) (2026)

## Abstract (arXiv)

Chain-of-thought (CoT) has emerged as a groundbreaking tool in NLP, notably for its efficacy in complex reasoning tasks, such as mathematical proofs. However, its application in code generation faces a distinct challenge, i.e., although the code generated with CoT reasoning is logically correct, it faces the problem of syntax error (e.g., invalid syntax error report) during code execution, which causes the CoT result's pass@1 in HumanEval even lower than the zero-shot result. In this paper, we present Code Chain-of-Thought (CodeCoT) that integrates CoT with a self-examination process for code generation. CodeCoT begins with the LLMs using CoT for initial code development to ensure the generated code follows the correct logic flow. Then, CodeCoT will generate test cases to validate whether the code has syntax errors during the execution. CodeCoT then employs a self-examination phase, in which the generated code is executed against these test cases in the local environment. If the local environment raises error information (e.g., invalid syntax error), CodeCoT will iteratively refine the code based on the feedback information. Within this loop, CodeCoT can make sure their generated codes not only follow the logic flow of the code description, but the syntax error will also be addressed with the self-examination process. Our evaluation results reveal that CodeCoT improves the effectiveness of code generation. For example, CodeCoT increases pass@1 from 75.6% to 79.3% for the HumanEval dataset.