# Automated Prompt Engineering for Cost-Effective Code Generation Using Evolutionary Algorithm

arXiv: https://arxiv.org/abs/2408.11198 · HTML: https://mlsanity.com/paper/2408.11198/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 47 references that are in the map.
- First posted on arXiv 20 Aug 2024.

## Builds on (most cited first, 30 of 32)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [Automatic Prompt Optimization with "Gradient Descent" and Beam Search](https://mlsanity.com/paper/2305.03495/) (2023)
- [Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models](https://mlsanity.com/paper/2310.04406/) (2023)
- [Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution](https://mlsanity.com/paper/2309.16797/) (2023)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications](https://mlsanity.com/paper/2402.07927/) (2024)
- [Self-collaboration Code Generation via ChatGPT](https://mlsanity.com/paper/2304.07590/) (2023)
- [Generated Knowledge Prompting for Commonsense Reasoning](https://mlsanity.com/paper/2110.08387/) (2021)
- [Software Testing with Large Language Models: Survey, Landscape, and Vision](https://mlsanity.com/paper/2307.07221/) (2023)
- [Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-step](https://mlsanity.com/paper/2402.16906/) (2024)
- [Chain-of-Knowledge: Grounding Large Language Models via Dynamic Knowledge Adapting over Heterogeneous Sources](https://mlsanity.com/paper/2305.13269/) (2023)
- [HyperPrompt: Prompt-based Task-Conditioning of Transformers](https://mlsanity.com/paper/2203.00759/) (2022)
- [Effective Test Generation Using Pre-trained Large Language Models and Mutation Testing](https://mlsanity.com/paper/2308.16557/) (2023)
- [Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency](https://mlsanity.com/paper/2309.17272/) (2023)
- [ClarifyGPT: Empowering LLM-based Code Generation with Intention Clarification](https://mlsanity.com/paper/2310.10996/) (2023)
- [On the Worst Prompt Performance of Large Language Models](https://mlsanity.com/paper/2406.10248/) (2024)
- [Can LLMs Generate Architectural Design Decisions? -An Exploratory Empirical study](https://mlsanity.com/paper/2403.01709/) (2024)
- [InstOptima: Evolutionary Multi-objective Instruction Optimization via Large Language Model-based Instruction Operators](https://mlsanity.com/paper/2310.17630/) (2023)
- [SPELL: Semantic Prompt Evolution based on a LLM](https://mlsanity.com/paper/2310.01260/) (2023)

## Built on by (most cited first, 5 of 5)

- [Can We Afford The Perfect Prompt? Balancing Cost and Accuracy with the Economical Prompting Index](https://mlsanity.com/paper/2412.01690/) (2024)
- [Can LLMs Replace Humans During Code Chunking?](https://mlsanity.com/paper/2506.19897/) (2025)
- [Prompt engineering and framework: implementation to increase code reliability based guideline for LLMs](https://mlsanity.com/paper/2506.10989/) (2025)
- [Search-Based Software Engineering and AI Foundation Models: Current Landscape and Future Roadmap](https://mlsanity.com/paper/2505.19625/) (2025)
- [Do Influence Tactics Matter? Investigating Prompt Framing Effects in LLM Code Generation](https://mlsanity.com/paper/2608.11513/) (2026)

## Abstract (arXiv)

Large Language Models have seen increasing use in various software development tasks, especially in code generation. The most advanced recent methods attempt to incorporate feedback from code execution into prompts to help guide LLMs in generating correct code in an iterative process. While effective, these methods could be costly due to numerous interactions with the LLM and extensive token usage. To address this issue, we propose an alternative approach named Evolutionary Prompt Engineering for Code (EPiC), which leverages a lightweight evolutionary algorithm to refine the original prompts into improved versions that generate high quality code, with minimal interactions with the LLM. Our evaluation against state-of-the-art (SOTA) LLM based code generation agents shows that EPiC not only achieves up to 6% improvement in pass@k but is also 2-10 times more cost-effective than the baselines.