mlsanityData to 14 Oct 2026

What Makes Large Language Models Reason in (Multi-Turn) Code Generation?

arXiv 2410.08105 ↗First posted 10 Oct 2024cs.CLcited by 29

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Prompting techniques such as chain-of-thought have established themselves as a popular vehicle for improving the outputs of large language models (LLMs). For code generation, however, their exact mechanics and efficacy are under-explored. We thus investigate the effects of a wide range of prompting strategies with a focus on automatic re-prompting over multiple turns and computational requirements. After systematically decomposing reasoning, instruction, and execution feedback prompts, we conduct an extensive grid search on the competitive programming benchmarks CodeContests and TACO for multiple LLM families and sizes (Llama 3.0 and 3.1, 8B, 70B, 405B, and GPT-4o). Our study reveals strategies that consistently improve performance across all models with small and large sampling budgets. We then show how finetuning with such an optimal configuration allows models to internalize the induced reasoning process and obtain improvements in performance and scalability for multi-turn code generation.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 48 of its 53 references that are in the map. First posted on arXiv 10 Oct 2024.

Builds on

48 of 53 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024MapCoder: Multi-Agent Code Generation for Competitive Problem Solving18 May
  2. 2024NExT: Teaching Large Language Models to Reason about Code Execution23 Apr
  3. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code12 Mar
  4. 2023Teaching Large Language Models to Self-Debug11 Apr
  5. 2022Competition-Level Code Generation with AlphaCode8 Feb

Other references · 43

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  6. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  8. 2021Program Synthesis with Large Language Modelscited by 3,637
  9. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  12. 2019The Curious Case of Neural Text Degenerationcited by 2,916
31 more
  1. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  2. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  3. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  4. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  5. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  6. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  7. 2021Measuring Coding Challenge Competence With APPScited by 902
  8. 2022PAL: Program-aided Language Modelscited by 652
  9. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 558
  10. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
  11. 2022CodeT: Code Generation with Generated Testscited by 384
  12. 2023Think before you speak: Training Language Models With Pause Tokenscited by 264
  13. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 252
  14. 2024Iterative Reasoning Preference Optimizationcited by 246
  15. 2024AI Agents That Mattercited by 219
  16. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 217
  17. 2023REFINER: Reasoning Feedback on Intermediate Representationscited by 205
  18. 2024Distilling System 2 into System 1cited by 159
  19. 2022Natural Language to Code Translation with Executioncited by 124
  20. 2024Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineeringcited by 103
  21. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  22. 2022Parsel: Algorithmic Reasoning with Language Models by Composing Decompositionscited by 64
  23. 2023CodeChain: Towards Modular Code Generation Through Chain of Self-revisions with Representative Sub-modulescited by 58
  24. 2024Code Repair with LLMs gives an Exploration-Exploitation Tradeoffcited by 51
  25. 2024The Larger the Better? Improved LLM Code-Generation via Budget Reallocationcited by 43
  26. 2024Efficient Tool Use with Chain-of-Abstraction Reasoningcited by 33
  27. 2024Critic-CoT: Boosting the reasoning abilities of large language model via Chain-of-thoughts Criticcited by 28
  28. 2023LLM-Assisted Code Cleaning For Training Accurate Code Generatorscited by 27
  29. 2024The Counterfeit Conundrum: Can Code Language Models Grasp the Nuances of Their Incorrect Generations?cited by 20
  30. 2024Investigating Mysteries of CoT-Augmented Distillationcited by 16
  31. 2023Explaining Competitive-Level Programming Solutions using LLMscited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 22; 2026: 5.

29 papers.

Sort:

By area:

  1. 2025Optimizing Language Models for Inference Time Objectives using Reinforcement Learningcited by 53
  2. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47
  3. 2025Teaching Language Models to Critique via Reinforcement Learningcited by 42
  4. 2024Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generationcited by 21
  5. 2025A Survey on LLM Test-Time Compute via Search: Tasks, LLM Profiling, Search Algorithms, and Relevant Frameworkscited by 21
  6. 2025debug-gym: A Text-Based Environment for Interactive Debuggingcited by 19
  7. 2025MAPRO: Recasting Multi-Agent Prompt Optimization as Maximum a Posteriori Inferencecited by 17
  8. 2025Towards Spoken Mathematical Reasoning: Benchmarking Speech-based Models over Multi-faceted Math Problemscited by 12
  9. 2025Iterative Self-Incentivization Empowers Large Language Models as Agentic Searcherscited by 11
  10. 2025RedCoder: Automated Multi-Turn Red Teaming for Code LLMscited by 9
  11. 2025Improving Diversity in Language Models: When Temperature Fails, Change the Losscited by 7
  12. 2026Mitigating Conversational Inertia in Multi-Turn Agentscited by 7
17 more
  1. 2025Large Language Models for Interpretable Mental Health Diagnosiscited by 6
  2. 2025The KoLMogorov Test: Compression by Code Generationcited by 5
  3. 2025FreshBrew: A Benchmark for Evaluating AI Agents on Java Code Migrationcited by 5
  4. 2025Self-Correcting Code Generation Using Small Language Modelscited by 4
  5. 2025Benchmarking Correctness and Security in Multi-Turn Code Generationcited by 4
  6. 2026Falsification, Not Exposure: An Internally Preregistered Placebo-Controlled Decomposition of Self-Repair Feedback in Frozen Small Code Modelscited by 4
  7. 2026Self-Execution Simulation Improves Coding Modelscited by 3
  8. 2024Evolutionary Pre-Prompt Optimization for Mathematical Reasoningcited by 2
  9. 2025Intention Chain-of-Thought Prompting with Dynamic Routing for Code Generationcited by 2
  10. 2025Tuning without Peeking: Provable Generalization Bounds and Robust LLM Post-Trainingcited by 1
  11. 2025Breaking the Myth: Can Small Models Infer Postconditions Too?cited by 1
  12. 2025SELF-REDRAFT: Eliciting Intrinsic Exploration-Exploitation Balance in Test-Time Scaling for Code Generationcited by 1
  13. 2025RPM-MCTS: Knowledge-Retrieval as Process Reward Model with Monte Carlo Tree Search for Code Generationcited by 1
  14. 2026Towards a Neural Debugger for Pythoncited by 1
  15. 2026Beyond pass@k: Redundancy-Aware RLVR for Multi-Sample Code Generationcited by 1
  16. 2025SAGE:Specification-Aware Grammar Extraction for Automated Test Case Generation with LLMscited by 0
  17. 2025code_transformed: The Influence of Large Language Models on Codecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhancing Code Generation Performance of Smaller Models by Distilling the Reasoning Ability of LLMs
  2. 2025Evaluating and Improving Large Language Models for Competitive Program Generation
  3. 2026VeriInteresting: An Empirical Study of Model Prompt Interactions in Verilog Code Generation
  4. 2025An Empirical Study of Reasoning Steps in Thinking Code LLMs
  5. 2024The Impact of Prompt Programming on Function-Level Code Generation
  6. 2025Prompt engineering and framework: implementation to increase code reliability based guideline for LLMs
  7. 2025Chain-of-Code Collapse: Reasoning Failures in LLMs via Adversarial Prompting in Code Generation
  8. 2025Understanding Chain-of-Thought Effectiveness in Code Generation: An Empirical and Information-Theoretic Analysis