mlsanityData to 14 Oct 2026

Structured Chain-of-Thought Prompting for Code Generation

arXiv 2305.06599 ↗First posted 11 May 2023cs.SE cs.CLcited by 138

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Large Language Models (LLMs) (e.g., ChatGPT) have shown impressive performance in code generation. LLMs take prompts as inputs, and Chain-of-Thought (CoT) prompting is the state-of-the-art prompting technique. CoT prompting asks LLMs first to generate CoTs (i.e., intermediate natural language reasoning steps) and then output the code. However, CoT prompting is designed for natural language generation and has low accuracy in code generation. In this paper, we propose Structured CoTs (SCoTs) and present a novel prompting technique for code generation, named SCoT prompting. Our motivation is source code contains rich structural information and any code can be composed of three program structures (i.e., sequence, branch, and loop structures). Intuitively, structured intermediate reasoning steps make for structured source code. Thus, we ask LLMs to use program structures to build CoTs, obtaining SCoTs. Then, LLMs generate the final code based on SCoTs. Compared to CoT prompting, SCoT prompting explicitly constrains LLMs to think about how to solve requirements from the view of source code and further the performance of LLMs in code generation. We apply SCoT prompting to two LLMs (i.e., ChatGPT and Codex) and evaluate it on three benchmarks (i.e., HumanEval, MBPP, and MBCPP). (1) SCoT prompting outperforms the state-of-the-art baseline - CoT prompting by up to 13.79% in Pass@1. (2) Human evaluation shows human developers prefer programs from SCoT prompting. (3) SCoT prompting is robust to examples and achieves substantial improvements.

Key facts. Cited by 138 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 27 of its 53 references that are in the map. First posted on arXiv 11 May 2023.

Builds on

27 of 53 references.

References · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  8. 2021Program Synthesis with Large Language Modelscited by 3,637
  9. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  11. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  12. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
15 more
  1. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  2. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  3. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  4. 2023StarCoder: may the source be with you!cited by 799
  5. 2015Sequence Level Training with Recurrent Neural Networkscited by 576
  6. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 498
  7. 2022CodeT: Code Generation with Generated Testscited by 384
  8. 2022Multi-lingual Evaluation of Code Generation Modelscited by 159
  9. 2023Self-Edit: Fault-Aware Code Editor for Code Generationcited by 111
  10. 2023What Makes Good In-context Demonstrations for Code Intelligence Tasks with LLMs?cited by 47
  11. 2023SkCoder: A Sketch-based Approach for Automatic Code Generationcited by 40
  12. 2022Fault-Aware Neural Code Rankerscited by 33
  13. 2022AixBench: A Code Generation Benchmark Datasetcited by 22
  14. 2022CodeEditor: Learning to Edit Source Code with Pre-trained Modelscited by 12
  15. 2021Modular Tree Network for Source Code Representation Learningcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 35; 2025: 60; 2026: 35.

138 papers.

Sort:

By area:

  1. 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applicationscited by 431
  2. 2025VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Modelscited by 107
  3. 2024Faithful Logical Reasoning via Symbolic Chain-of-Thoughtcited by 100
  4. 2024EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositoriescited by 72
  5. 2024DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositoriescited by 72
  6. 2025AgentRxiv: Towards Collaborative Autonomous Researchcited by 72
  7. 2024Vul-RAG: Enhancing LLM-based Vulnerability Detection via Knowledge-level RAGcited by 68
  8. 2024Bugs in Large Language Models Generated Code: An Empirical Studycited by 60
  9. 2024Agents in Software Engineering: Survey, Landscape, and Visioncited by 60
  10. 2024Exploring and Unleashing the Power of Large Language Models in Automated Code Translationcited by 57
  11. 2025Large Language Models Hallucination: A Comprehensive Surveycited by 49
  12. 2024A Survey of Prompt Engineering Methods in Large Language Models for Different NLP Taskscited by 47
38 more
  1. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47
  2. 2023Large Language Model-Aware In-Context Learning for Code Generationcited by 43
  3. 2023Hot or Cold? Adaptive Temperature Sampling for Code Generation with Large Language Modelscited by 38
  4. 2023AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisationcited by 35
  5. 2024SOEN-101: Code Generation by Emulating Software Process Models Using Large Language Model Agentscited by 34
  6. 2024EvoCodeBench: An Evolving Code Generation Benchmark with Domain-Specific Evaluationscited by 34
  7. 2025CodeCoR: An LLM-Based Self-Reflective Multi-Agent Framework for Code Generationcited by 34
  8. 2024Text-Tuple-Table: Towards Information Integration in Text-to-Table Generation via Global Tuple Extractioncited by 33
  9. 2025Uncertainty-Guided Chain-of-Thought for Code Generation with LLMscited by 32
  10. 2024Benchmarking Data Science Agentscited by 30
  11. 2023CodeCoT: Tackling Code Syntax Errors in CoT Reasoning for Code Generationcited by 29
  12. 2025AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunitiescited by 28
  13. 2023Chain-of-Thought in Neural Code Generation: From and For Lightweight Language Modelscited by 27
  14. 2025Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settingscited by 24
  15. 2025SEW: Self-Evolving Agentic Workflows for Automated Code Generationcited by 20
  16. 2023INTERVENOR: Prompting the Coding Ability of Large Language Models with the Interactive Chain of Repaircited by 18
  17. 2024UniCoder: Scaling Code Large Language Model via Universal Codecited by 18
  18. 2024Evaluating Software Development Agents: Patch Patterns, Code Quality, and Issue Complexity in Real-World GitHub Scenarioscited by 18
  19. 2024OMPGPT: A Generative Pre-trained Transformer Model for OpenMPcited by 16
  20. 2025Could Thinking Multilingually Empower LLM Reasoning?cited by 16
  21. 2025Give LLMs a Security Course: Securing Retrieval-Augmented Code Generation via Knowledge Injectioncited by 16
  22. 2025Exploring the Potential of Diffusion Large Language Models in Code Generationcited by 16
  23. 2024Leveraging Print Debugging to Improve Code Generation in Large Language Modelscited by 15
  24. 2024Beyond Functional Correctness: Investigating Coding Style Inconsistencies in Large Language Modelscited by 15
  25. 2025Are Large Language Models Reliable AI Scientists? Assessing Reverse-Engineering of Black-Box Systemscited by 14
  26. 2024Leveraging Large Language Models for Enhancing the Understandability of Generated Unit Testscited by 11
  27. 2025Enhancing LLM-based Quantum Code Generation with Multi-Agent Optimization and Quantum Error Correctioncited by 11
  28. 2025Lang-PINN: From Language to Physics-Informed Neural Networks via a Multi-Agent Frameworkcited by 11
  29. 2025Exploring Code Language Models for Automated HLS-based Hardware Generation: Benchmark, Infrastructure and Analysiscited by 10
  30. 2025Rethinking the Evaluation of Secure Code Generationcited by 10
  31. 2024DevEval: Evaluating Code Generation in Practical Software Projectscited by 9
  32. 2025Can LLMs Generate Reliable Test Case Generators? A Study on Competition-Level Programming Problemscited by 9
  33. 2025Automated Knowledge Graph Construction using Large Language Models and Sentence Complexity Modellingcited by 9
  34. 2024Say More with Less: Understanding Prompt Learning Behaviors through Gist Compressioncited by 8
  35. 2025ARCS: Agentic Retrieval-Augmented Code Synthesis with Iterative Refinementcited by 7
  36. 2025RedTeamLLM: an Agentic AI framework for offensive securitycited by 7
  37. 2025What Makes a Good Natural Language Prompt?cited by 7
  38. 2023Bridging Code Semantic and LLMs: Semantic Chain-of-Thought Prompting for Code Generationcited by 6

50 most cited of 138 All 138 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Understanding Chain-of-Thought Effectiveness in Code Generation: An Empirical and Information-Theoretic Analysis
  2. 2025Are They All Good? Evaluating the Quality of CoTs in LLM-based Code Generation
  3. 2023Towards Better Chain-of-Thought Prompting Strategies: A Survey
  4. 2023Chain-Of-Thought Prompting Under Streaming Batch: A Case Study
  5. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters
  6. 2022Automatic Chain of Thought Prompting in Large Language Models
  7. 2025Prompt engineering and framework: implementation to increase code reliability based guideline for LLMs
  8. 2025MSCoT: Structured Chain-of-Thought Generation for Multiple Programming Languages