# Search-Based LLMs for Code Optimization

arXiv: https://arxiv.org/abs/2408.12159 · HTML: https://mlsanity.com/paper/2408.12159/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 55 references that are in the map.
- First posted on arXiv 22 Aug 2024.

## Builds on (most cited first, 30 of 32)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Large Language Models Cannot Self-Correct Reasoning Yet](https://mlsanity.com/paper/2310.01798/) (2023)
- [Atlas: Few-shot Learning with Retrieval Augmented Language Models](https://mlsanity.com/paper/2208.03299/) (2022)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [StarCoder: may the source be with you!](https://mlsanity.com/paper/2305.06161/) (2023)
- [WizardCoder: Empowering Code Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2306.08568/) (2023)
- [CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks](https://mlsanity.com/paper/2105.12655/) (2021)
- [Planning with Large Language Models for Code Generation](https://mlsanity.com/paper/2303.05510/) (2023)
- [Practical Program Repair in the Era of Large Pre-trained Language Models](https://mlsanity.com/paper/2210.14179/) (2022)
- [Self-Edit: Fault-Aware Code Editor for Code Generation](https://mlsanity.com/paper/2305.04087/) (2023)
- [Repository-Level Prompt Generation for Large Language Models of Code](https://mlsanity.com/paper/2206.12839/) (2022)
- [Learning Performance-Improving Code Edits](https://mlsanity.com/paper/2302.07867/) (2023)
- [A Syntax-Guided Edit Decoder for Neural Program Repair](https://mlsanity.com/paper/2106.08253/) (2021)
- [What Makes Good In-context Demonstrations for Code Intelligence Tasks with LLMs?](https://mlsanity.com/paper/2304.07575/) (2023)
- [Code Structure Guided Transformer for Source Code Summarization](https://mlsanity.com/paper/2104.09340/) (2021)
- [Domain Knowledge Matters: Improving Prompts with Fix Templates for Repairing Python Type Errors](https://mlsanity.com/paper/2306.01394/) (2023)
- [ITER: Iterative Neural Repair for Multi-Location Patches](https://mlsanity.com/paper/2304.12015/) (2023)
- [Supersonic: Learning to Generate Source Code Optimizations in C/C++](https://mlsanity.com/paper/2309.14846/) (2023)
- [RAPGen: An Approach for Fixing Code Inefficiencies in Zero-Shot](https://mlsanity.com/paper/2306.17077/) (2023)
- [Enhancing Genetic Improvement Mutations Using Large Language Models](https://mlsanity.com/paper/2310.19813/) (2023)
- [Learning in the Wild: Towards Leveraging Unlabeled Data for Effectively Tuning Pre-trained Code Models](https://mlsanity.com/paper/2401.01060/) (2024)
- [Generative Type Inference for Python](https://mlsanity.com/paper/2307.09163/) (2023)
- [Triangulating Python Performance Issues with Scalene](https://mlsanity.com/paper/2212.07597/) (2022)
- [Search-based Optimisation of LLM Learning Shots for Story Point Estimation](https://mlsanity.com/paper/2403.08430/) (2024)

## Built on by (most cited first, 28 of 28)

- [Agent READMEs: An Empirical Study of Context Files for Agentic Coding](https://mlsanity.com/paper/2511.12884/) (2025)
- [SWE-MeM: Learning Adaptive Memory Management for Long-Horizon Coding Agents](https://mlsanity.com/paper/2606.28434/) (2026)
- [Multi-Programming Language Sandbox for LLMs](https://mlsanity.com/paper/2410.23074/) (2024)
- [CoopetitiveV: Leveraging LLM-powered Coopetitive Multi-Agent Prompting for High-quality Verilog Generation](https://mlsanity.com/paper/2412.11014/) (2024)
- [EffiSkill: Agent Skill Based Automated Code Efficiency Optimization](https://mlsanity.com/paper/2603.27850/) (2026)
- [Knowledge Augmented Complex Problem Solving with Large Language Models: A Survey](https://mlsanity.com/paper/2505.03418/) (2025)
- [RobuNFR: Evaluating the Robustness of Large Language Models on Non-Functional Requirements Aware Code Generation](https://mlsanity.com/paper/2503.22851/) (2025)
- [RAG or Fine-tuning? A Comparative Study on LCMs-based Code Completion in Industry](https://mlsanity.com/paper/2505.15179/) (2025)
- [LLM Program Optimization via Retrieval Augmented Search](https://mlsanity.com/paper/2501.18916/) (2025)
- [FALCON: Transforming Cyber Threat Intelligence into Deployable IDS Rules with Self-Reflection](https://mlsanity.com/paper/2508.18684/) (2025)
- [AutoTrainess: Teaching Language Models to Improve Language Models Autonomously](https://mlsanity.com/paper/2606.31551/) (2026)
- [PerfCoder: Large Language Models for Interpretable Code Performance Optimization](https://mlsanity.com/paper/2512.14018/) (2025)
- [FasterPy: An LLM-based Code Execution Efficiency Optimization Framework](https://mlsanity.com/paper/2512.22827/) (2025)
- [PerfAgent: Profiler-Guided Iterative Refinement for Repository-Level Code Optimization](https://mlsanity.com/paper/2607.19653/) (2026)
- [ECO: Enhanced Code Optimization via Performance-Aware Prompting for Code-LLMs](https://mlsanity.com/paper/2510.10517/) (2025)
- [Optimizing Code Runtime Performance through Context-Aware Retrieval-Augmented Generation](https://mlsanity.com/paper/2501.16692/) (2025)
- [Investigating Execution-Aware Language Models for Code Optimization](https://mlsanity.com/paper/2503.08228/) (2025)
- [A Blueprint for AI-Driven Software Quality: Integrating LLMs with Established Standards](https://mlsanity.com/paper/2505.13766/) (2025)
- [Toward Automated and Trustworthy Scientific Analysis and Visualization with LLM-Generated Code](https://mlsanity.com/paper/2511.21920/) (2025)
- [LOOPRAG: Enhancing Loop Transformation Optimization with Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2512.15766/) (2025)
- [AutoPass: Evidence-Guided LLM Agents for Compiler Performance Tuning](https://mlsanity.com/paper/2606.20373/) (2026)
- [Search-Based Software Engineering and AI Foundation Models: Current Landscape and Future Roadmap](https://mlsanity.com/paper/2505.19625/) (2025)
- [GrowthHacker: Automated Off-Policy Evaluation Optimization Using Code-Modifying LLM Agents](https://mlsanity.com/paper/2511.00802/) (2025)
- [The New Compiler Stack: A Survey on the Synergy of LLMs and Compilers](https://mlsanity.com/paper/2601.02045/) (2026)
- [Coding in a Bubble? Evaluating LLMs in Resolving Context Adaptation Bugs During Code Adaptation](https://mlsanity.com/paper/2601.06497/) (2026)
- [ConceptCoder: Improve Code Reasoning via Concept Learning](https://mlsanity.com/paper/2603.23470/) (2026)
- [Can Coding Agents Implement Missed Compiler Optimizations? Evaluating LLM Agents on LLVM Peephole Optimizations](https://mlsanity.com/paper/2607.02684/) (2026)
- [Reinforcement Learning for Code Optimization](https://mlsanity.com/paper/2607.25970/) (2026)

## Abstract (arXiv)

The code written by developers usually suffers from efficiency problems and contain various performance bugs. These inefficiencies necessitate the research of automated refactoring methods for code optimization. Early research in code optimization employs rule-based methods and focuses on specific inefficiency issues, which are labor-intensive and suffer from the low coverage issue. Recent work regards the task as a sequence generation problem, and resorts to deep learning (DL) techniques such as large language models (LLMs). These methods typically prompt LLMs to directly generate optimized code. Although these methods show state-of-the-art performance, such one-step generation paradigm is hard to achieve an optimal solution. First, complex optimization methods such as combinatorial ones are hard to be captured by LLMs. Second, the one-step generation paradigm poses challenge in precisely infusing the knowledge required for effective code optimization within LLMs, resulting in under-optimized code.To address these problems, we propose to model this task from the search perspective, and propose a search-based LLMs framework named SBLLM that enables iterative refinement and discovery of improved optimization methods. SBLLM synergistically integrate LLMs with evolutionary search and consists of three key components: 1) an execution-based representative sample selection part that evaluates the fitness of each existing optimized code and prioritizes promising ones to pilot the generation of improved code; 2) an adaptive optimization pattern retrieval part that infuses targeted optimization patterns into the model for guiding LLMs towards rectifying and progressively enhancing their optimization methods; and 3) a genetic operator-inspired chain-of-thought prompting part that aids LLMs in combining different optimization methods and generating improved optimization methods.