# Prompt Alchemy: Automatic Prompt Refinement for Enhancing Code Generation

arXiv: https://arxiv.org/abs/2503.11085 · HTML: https://mlsanity.com/paper/2503.11085/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Mar 2025.

## Built on by (most cited first, 8 of 8)

- [A Survey of Context Engineering for Large Language Models](https://mlsanity.com/paper/2507.13334/) (2025)
- [OptiML: An End-to-End Framework for Program Synthesis and CUDA Kernel Optimization](https://mlsanity.com/paper/2602.12305/) (2026)
- [GitChameleon 2.0: Evaluating AI Code Generation Against Python Library Version Incompatibilities](https://mlsanity.com/paper/2507.12367/) (2025)
- [VibeCodeHPC: An Agent-Based Iterative Prompting Auto-Tuner for HPC Code Generation Using LLMs](https://mlsanity.com/paper/2510.00031/) (2025)
- [PrismVAU: Prompt-Refined Inference System for Multimodal Video Anomaly Understanding](https://mlsanity.com/paper/2601.02927/) (2026)
- [VeriInteresting: An Empirical Study of Model Prompt Interactions in Verilog Code Generation](https://mlsanity.com/paper/2603.08715/) (2026)
- [QiMeng-NeuComBack: Self-Evolving Translation from IR to Assembly Code](https://mlsanity.com/paper/2511.01183/) (2025)
- [Self-Evolving Coding Rules for AI Coding Agents](https://mlsanity.com/paper/2610.00650/) (2026)

## Abstract (arXiv)

Code generation has emerged as a key task to automate software development by converting high-level descriptions into executable code. Large language models (LLMs) excel at this but depend heavily on input prompt quality.Manual prompt engineering can be time-consuming and inconsistent, limiting LLM effectiveness. This paper introduces Prochemy, an innovative method for automatically refining prompts to boost code generation. Prochemy overcomes manual prompt limitations by automating optimization, ensuring consistency during inference, and supporting multi-agent systems.It iteratively refines prompts based on model performance, using an optimized final prompt for improved consistency across tasks. We tested Prochemy on natural language-based code generation and translation tasks using three LLM series. Results indicate Prochemy enhances existing methods, improving performance by 5.0% for GPT-3.5-Turbo and 1.9% for GPT-4o over zero-shot baselines on HumanEval. In state-of-the-art LDB, Prochemy + LDB surpasses standalone methods by 1.2-1.8%. For code translation, Prochemy boosts GPT-4o's Java-to-Python (AVATAR) performance from 74.5 to 84.1 (+12.9%) and Python-to-Java from 66.8 to 78.2 (+17.1%). Moreover, Prochemy maintains strong performance when integrated with the o1-mini model, validating its efficacy in code tasks. Designed as plug-and-play, Prochemy optimizes prompts with minimal human input, bridging the gap between simple prompts and complex frameworks.