# Chain-of-Thought in Neural Code Generation: From and For Lightweight Language Models

arXiv: https://arxiv.org/abs/2312.05562 · HTML: https://mlsanity.com/paper/2312.05562/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 14 of its 109 references that are in the map.
- First posted on arXiv 9 Dec 2023.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 14 of 14)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [Towards Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2212.10403/) (2022)
- [Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning](https://mlsanity.com/paper/2205.09712/) (2022)
- [Large Language Models Are Reasoning Teachers](https://mlsanity.com/paper/2212.10071/) (2022)
- [Reasoning with Language Model Prompting: A Survey](https://mlsanity.com/paper/2212.09597/) (2022)
- [Specializing Smaller Language Models towards Multi-Step Reasoning](https://mlsanity.com/paper/2301.12726/) (2023)
- [Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step](https://mlsanity.com/paper/2306.14050/) (2023)
- [Rethinking with Retrieval: Faithful Large Language Model Inference](https://mlsanity.com/paper/2301.00303/) (2022)
- [Structured Chain-of-Thought Prompting for Code Generation](https://mlsanity.com/paper/2305.06599/) (2023)

## Built on by (most cited first, 27 of 27)

- [Making Reasoning Matter: Measuring and Improving Faithfulness of Chain-of-Thought Reasoning](https://mlsanity.com/paper/2402.13950/) (2024)
- [Agents in Software Engineering: Survey, Landscape, and Vision](https://mlsanity.com/paper/2409.09030/) (2024)
- [Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMs](https://mlsanity.com/paper/2502.19411/) (2025)
- [Chain-of-Thought Prompting Obscures Hallucination Cues in Large Language Models: An Empirical Evaluation](https://mlsanity.com/paper/2506.17088/) (2025)
- [VeriThoughts: Enabling Automated Verilog Code Generation using Reasoning and Formal Verification](https://mlsanity.com/paper/2505.20302/) (2025)
- [Issue-Oriented Agent-Based Framework for Automated Review Comment Generation](https://mlsanity.com/paper/2511.00517/) (2025)
- [Falsification, Not Exposure: An Internally Preregistered Placebo-Controlled Decomposition of Self-Repair Feedback in Frozen Small Code Models](https://mlsanity.com/paper/2606.31511/) (2026)
- [Designing LLM-based Multi-Agent Systems for Software Engineering Tasks: Quality Attributes, Design Patterns and Rationale](https://mlsanity.com/paper/2511.08475/) (2025)
- [Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Models](https://mlsanity.com/paper/2603.13985/) (2026)
- [CrystalReasoner: Reasoning and RL for Property-Conditioned Crystal Structure Generation](https://mlsanity.com/paper/2605.14344/) (2026)
- [Boost, Disentangle, and Customize: A Robust System2-to-System1 Pipeline for Code Generation](https://mlsanity.com/paper/2502.12492/) (2025)
- [Optimizing Token Consumption in LLMs: A Nano Surge Approach for Code Reasoning Efficiency](https://mlsanity.com/paper/2504.15989/) (2025)
- [Adversarial Reasoning for Repair Based on Inferred Program Intent](https://mlsanity.com/paper/2505.13008/) (2025)
- [CODE-DITING: A Reasoning-Based Metric for Functional Alignment in Code Evaluation](https://mlsanity.com/paper/2505.19502/) (2025)
- [Rethinking Reasoning: A Survey on Reasoning-based Backdoors in LLMs](https://mlsanity.com/paper/2510.07697/) (2025)
- [Code-Driven Inductive Synthesis: Enhancing Reasoning Abilities of Large Language Models with Sequences](https://mlsanity.com/paper/2503.13109/) (2025)
- [The Cream Rises to the Top: Efficient Reranking Method for Verilog Code Generation](https://mlsanity.com/paper/2509.20215/) (2025)
- [Unseen-Codebases-Domain Data Synthesis and Training Based on Code Graphs](https://mlsanity.com/paper/2602.20799/) (2026)
- [Can LLM Aid in Solving Constraints with Inductive Definitions?](https://mlsanity.com/paper/2603.03668/) (2026)
- [Developing LLM-based Multi-Agent Systems in Software Engineering: A Mixed-Method Experience Report](https://mlsanity.com/paper/2608.11965/) (2026)
- [Toward Non-Expert Customized Congestion Control: Large Language Model-Assisted CCA Code Generation with eBPF Deployment](https://mlsanity.com/paper/2601.22461/) (2026)
- [Design-Specification Tiling for ICL-based CAD Code Generation](https://mlsanity.com/paper/2603.12712/) (2026)
- [Temperature-Dependent Performance of Prompting Strategies in Extended Reasoning Large Language Models](https://mlsanity.com/paper/2604.08563/) (2026)
- [CollabCoder: Plan-Code Co-Evolution via Collaborative Decision-Making for Efficient Code Generation](https://mlsanity.com/paper/2604.13946/) (2026)
- [Improving LLM Code Generation via Requirement-Aware Curriculum Reinforcement Learning](https://mlsanity.com/paper/2605.00433/) (2026)
- [Repository-Level Solidity Code Generation with Large Language Models: From Prompting to Fine-Tuning](https://mlsanity.com/paper/2606.19988/) (2026)
- [CIT-CAD: Constraint Intent Tree-based CAD Code Generation and Verification](https://mlsanity.com/paper/2609.07434/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable potential in code generation. The integration of Chain of Thought (CoT) reasoning can further boost their performance. However, current CoT methods often require manual writing or LLMs with over 100 billion parameters to generate, impeding their applicability in resource-constrained scenarios. In this study, we investigate lightweight Language Models (lLMs), which are defined to have fewer than 10 billion parameters. Empirically, we find that most lLMs cannot generate high-quality CoTs when prompted by the few-shot method, but can take advantage of high-quality CoTs generated elsewhere to improve their performance in code generation. Based on these findings, we design a novel approach COTTON which can leverage lLMs to automatically generate CoTs for code generation. We synthesize new datasets and conduct extensive experiments on various benchmarks. The results show that the CoTs generated by COTTON outperform the baselines in terms of automated and human evaluation metrics. In particular, the CoTs generated by COTTON boost various lLMs to achieve higher performance gains than those generated by LLMs such as ChatGLM (130B), and are competitive with those generated by gpt-3.5-turbo (175B). Our study also showcases the potential of lLMs in software engineering applications.