mlsanityData to 14 Oct 2026

Planning In Natural Language Improves LLM Search For Code Generation

arXiv 2409.03733 ↗First posted 5 Sep 2024cs.LG cs.AI cs.CLcited by 83

Not on the map. Why?

Cited by 37 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

While scaling training compute has led to remarkable improvements in large language models (LLMs), scaling inference compute has not yet yielded analogous gains. We hypothesize that a core missing component is a lack of diverse LLM outputs, leading to inefficient search due to models repeatedly sampling highly similar, yet incorrect generations. We empirically demonstrate that this lack of diversity can be mitigated by searching over candidate plans for solving a problem in natural language. Based on this insight, we propose PlanSearch, a novel search algorithm which shows strong results across HumanEval+, MBPP+, and LiveCodeBench (a contamination-free benchmark for competitive coding). PlanSearch generates a diverse set of observations about the problem and then uses these observations to construct plans for solving the problem. By searching over plans in natural language rather than directly over code solutions, PlanSearch explores a significantly more diverse range of potential solutions compared to baseline search methods. Using PlanSearch on top of Claude 3.5 Sonnet achieves a state-of-the-art pass@200 of 77.0% on LiveCodeBench, outperforming both the best score achieved without search (pass@1 = 41.4%) and using standard repeated sampling (pass@200 = 60.6%). Finally, we show that, across all models, search algorithms, and benchmarks analyzed, we can accurately predict performance gains due to search as a direct function of the diversity over generated ideas. Code can be found at https://github.com/scaleapi/plansearch.

Key facts. Cited by 83 papers in the map (data through 14 Oct 2026). First posted on arXiv 5 Sep 2024.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 13; 2025: 46; 2026: 24.

83 papers.

Sort:

By area:

  1. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 511
  2. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 223
  3. 2024Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agentscited by 154
  4. 2024AutoGLM: Autonomous Foundation Agents for GUIscited by 100
  5. 2024SafeBench: A Safety Evaluation Framework for Multimodal Large Language Modelscited by 69
  6. 2024The Limits of Inference Scaling Through Resamplingcited by 53
  7. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  8. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47
  9. 2026Code as Agent Harnesscited by 47
  10. 2024DrugAgent: Automating AI-aided Drug Discovery Programming through LLM Multi-Agent Collaborationcited by 44
  11. 2025Agent-RLVR: Training Software Engineering Agents via Guidance and Environment Rewardscited by 42
  12. 2025LLMs as Planning Formalizers: A Survey for Leveraging Large Language Models to Construct Automated Planning Modelscited by 38
38 more
  1. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 35
  2. 2025Optimizing Temperature for Language Models with Multi-Sample Inferencecited by 33
  3. 2025RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Codecited by 30
  4. 2024RethinkMCTS: Refining Erroneous Thoughts in Monte Carlo Tree Search for Code Generationcited by 26
  5. 2024B-STaR: Monitoring and Balancing Exploration and Exploitation in Self-Taught Reasonerscited by 26
  6. 2025GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agentscited by 25
  7. 2025A Survey on LLM Test-Time Compute via Search: Tasks, LLM Profiling, Search Algorithms, and Relevant Frameworkscited by 21
  8. 2026Structured Scaling of AI Discovery Across Diverse Scientific Domainscited by 21
  9. 2025Self-Generated In-Context Examples Improve LLM Agents for Sequential Decision-Making Taskscited by 19
  10. 2025RPG: A Repository Planning Graph for Unified and Scalable Codebase Generationcited by 19
  11. 2025On the Effect of Sampling Diversity in Scaling LLM Inferencecited by 18
  12. 2025Rethinking Inference-Time Scaling: Efficiency Limits and Linguistic Signalscited by 18
  13. 2025CER: Confidence Enhanced Reasoning in LLMscited by 17
  14. 2025Large Language Models (LLMs) for Source Code Analysis: applications, models and datasetscited by 16
  15. 2025PlanGEN: A Multi-Agent Framework for Generating Planning and Reasoning Trajectories for Complex Problem Solvingcited by 15
  16. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 15
  17. 2025Teaching LLMs to Plan: Logical Chain-of-Thought Instruction Tuning for Symbolic Planningcited by 15
  18. 2025Optimizing Sequential Multi-Step Tasks with Parallel LLM Agentscited by 14
  19. 2024Scattered Forest Search: Smarter Code Space Exploration with LLMscited by 10
  20. 2024SRA-MCTS: Self-driven Reasoning Augmentation with Monte Carlo Tree Search for Code Generationcited by 10
  21. 2025GPO: Learning from Critical Steps to Improve LLM Reasoningcited by 10
  22. 2025DISC: Dynamic Decomposition Improves LLM Inference Scalingcited by 9
  23. 2025Rethinking the Sampling Criteria in Reinforcement Learning for LLM Reasoning: A Competence-Difficulty Alignment Perspectivecited by 9
  24. 2025Generalizable Heuristic Generation Through LLMs with Meta-Optimizationcited by 9
  25. 2025Large Reasoning Embedding Models: Towards Next-Generation Dense Retrieval Paradigmcited by 9
  26. 2025Dynamic Risk Assessments for Offensive Cybersecurity Agentscited by 8
  27. 2025JoyAgent-JDGenie: Technical Report on the GAIAcited by 7
  28. 2024Generating Diverse Hypotheses for Inductive Reasoningcited by 6
  29. 2025EvoMem: Improving Multi-Agent Planning with Dual-Evolving Memorycited by 6
  30. 2025Scaling Test-Time Inference with Policy-Optimized, Dynamic Retrieval-Augmented Generation via KV Caching and Decodingcited by 5
  31. 2025Extended Inductive Reasoning for Personalized Preference Inference from Behavioral Signalscited by 5
  32. 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlookcited by 5
  33. 2024Entropy Controllable Direct Preference Optimizationcited by 4
  34. 2025ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Groundingcited by 4
  35. 2025NL-Debugging: Exploiting Natural Language as an Intermediate Representation for Code Debuggingcited by 4
  36. 2025Code Reasoning for Software Engineering Tasks: A Survey and A Call to Actioncited by 4
  37. 2025Program Synthesis via Test-Time Transductioncited by 4
  38. 2025MetaMuse: Algorithm Generation via Creative Ideationcited by 4

50 most cited of 83 All 83 (JSON)