# Think Outside the Code: Brainstorming Boosts Large Language Models in Code Generation

arXiv: https://arxiv.org/abs/2305.10679 · HTML: https://mlsanity.com/paper/2305.10679/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 6 of its 47 references that are in the map.
- First posted on arXiv 18 May 2023.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 6 of 6)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [Parsel: Algorithmic Reasoning with Language Models by Composing Decompositions](https://mlsanity.com/paper/2212.10561/) (2022)

## Built on by (most cited first, 19 of 19)

- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [Agents in Software Engineering: Survey, Landscape, and Vision](https://mlsanity.com/paper/2409.09030/) (2024)
- [Evaluating LLMs' Divergent Thinking Capabilities for Scientific Idea Generation with Minimal Context](https://mlsanity.com/paper/2412.17596/) (2024)
- [LLM-Assisted Code Cleaning For Training Accurate Code Generators](https://mlsanity.com/paper/2311.14904/) (2023)
- [Learning to Reason via Mixture-of-Thought for Logical Reasoning](https://mlsanity.com/paper/2505.15817/) (2025)
- [SwarmBrain: Embodied agent for real-time strategy game StarCraft II via large language models](https://mlsanity.com/paper/2401.17749/) (2024)
- [Enhancing Code Generation Performance of Smaller Models by Distilling the Reasoning Ability of LLMs](https://mlsanity.com/paper/2403.13271/) (2024)
- [Leveraging Print Debugging to Improve Code Generation in Large Language Models](https://mlsanity.com/paper/2401.05319/) (2024)
- [Beyond Functional Correctness: Investigating Coding Style Inconsistencies in Large Language Models](https://mlsanity.com/paper/2407.00456/) (2024)
- [Think Before You Speak: Cultivating Communication Skills of Large Language Models via Inner Monologue](https://mlsanity.com/paper/2311.07445/) (2023)
- [SRA-MCTS: Self-driven Reasoning Augmentation with Monte Carlo Tree Search for Code Generation](https://mlsanity.com/paper/2411.11053/) (2024)
- [ArchCode: Incorporating Software Requirements in Code Generation with Large Language Models](https://mlsanity.com/paper/2408.00994/) (2024)
- [A Survey on Large Language Models from Concept to Implementation](https://mlsanity.com/paper/2403.18969/) (2024)
- [RefleXGen:The unexamined code is not worth using](https://mlsanity.com/paper/2510.23674/) (2025)
- [Yggdrasil: Bridging Dynamic Speculation and Static Runtime for Latency-Optimal Tree-Based LLM Decoding](https://mlsanity.com/paper/2512.23858/) (2025)
- [Comments as Natural Logic Pivots: Improve Code Generation via Comment Perspective](https://mlsanity.com/paper/2404.07549/) (2024)
- [Weaker LLMs' Opinions Also Matter: Mixture of Opinions Enhances LLM's Mathematical Reasoning](https://mlsanity.com/paper/2502.19622/) (2025)
- [PatternGPT :A Pattern-Driven Framework for Large Language Model Text Generation](https://mlsanity.com/paper/2307.00470/) (2023)
- [Evaluating ChatGPT-3.5 Efficiency in Solving Coding Problems of Different Complexity Levels: An Empirical Analysis](https://mlsanity.com/paper/2411.07529/) (2024)

## Abstract (arXiv)

Code generation aims to automatically generate source code from high-level task specifications, which can significantly increase productivity of software engineering. Recently, approaches based on large language models (LLMs) have shown remarkable code generation abilities on simple tasks. However, generate code for more complex tasks, such as competition-level problems, remains challenging. In this paper, we introduce Brainstorm framework for code generation. It leverages a brainstorming step that generates and selects diverse thoughts on the problem to facilitate algorithmic reasoning, where the thoughts are possible blueprint of solving the problem. We demonstrate that Brainstorm significantly enhances the ability of LLMs to solve competition-level programming problems, resulting in a more than 50% increase in the pass@k metrics for ChatGPT on the CodeContests benchmark, achieving state-of-the-art performance. Furthermore, our experiments conducted on LeetCode contests show that our framework boosts the ability of ChatGPT to a level comparable to that of human programmers.