# Are Large Language Models Good Prompt Optimizers?

arXiv: https://arxiv.org/abs/2402.02101 · HTML: https://mlsanity.com/paper/2402.02101/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 50 references that are in the map.
- First posted on arXiv 3 Feb 2024.

## Builds on (most cited first, 30 of 39)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [Large Language Models Cannot Self-Correct Reasoning Yet](https://mlsanity.com/paper/2310.01798/) (2023)
- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023)
- [Automatic Prompt Optimization with "Gradient Descent" and Beam Search](https://mlsanity.com/paper/2305.03495/) (2023)
- [RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning](https://mlsanity.com/paper/2205.12548/) (2022)
- [Do Prompt-Based Models Really Understand the Meaning of their Prompts?](https://mlsanity.com/paper/2109.01247/) (2021)
- [Evaluating Large Language Models at Evaluating Instruction Following](https://mlsanity.com/paper/2310.07641/) (2023)
- [PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization](https://mlsanity.com/paper/2310.16427/) (2023)
- [GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Models](https://mlsanity.com/paper/2203.07281/) (2022)
- [Black-Box Prompt Optimization: Aligning Large Language Models without Model Training](https://mlsanity.com/paper/2311.04155/) (2023)
- [Can Large Language Models Really Improve by Self-critiquing Their Own Plans?](https://mlsanity.com/paper/2310.08118/) (2023)

## Built on by (most cited first, 30 of 37)

- [AgentSquare: Automatic LLM Agent Search in Modular Design Space](https://mlsanity.com/paper/2410.06153/) (2024)
- [Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems](https://mlsanity.com/paper/2504.01990/) (2025)
- [X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agents](https://mlsanity.com/paper/2504.13203/) (2025)
- [What Did I Do Wrong? Quantifying LLMs' Sensitivity and Consistency to Prompt Engineering](https://mlsanity.com/paper/2406.12334/) (2024)
- [Trace is the Next AutoDiff: Generative Optimization with Rich Feedback, Execution Traces, and LLMs](https://mlsanity.com/paper/2406.16218/) (2024)
- [A Systematic Survey on Large Language Models for Algorithm Design](https://mlsanity.com/paper/2410.14716/) (2024)
- [Efficient Prompting Methods for Large Language Models: A Survey](https://mlsanity.com/paper/2404.01077/) (2024)
- [S2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learning](https://mlsanity.com/paper/2502.12853/) (2025)
- [Aviary: training language agents on challenging scientific tasks](https://mlsanity.com/paper/2412.21154/) (2024)
- [Prompt-A-Video: Prompt Your Video Diffusion Model via Preference-Aligned LLM](https://mlsanity.com/paper/2412.15156/) (2024)
- [Entropy-Gated Branching for Efficient Test-Time Reasoning](https://mlsanity.com/paper/2503.21961/) (2025)
- [Efficient and Accurate Prompt Optimization: the Benefit of Memory in Exemplar-Guided Reflection](https://mlsanity.com/paper/2411.07446/) (2024)
- [On the Effect of Sampling Diversity in Scaling LLM Inference](https://mlsanity.com/paper/2502.11027/) (2025)
- [Verbalized Machine Learning: Revisiting Machine Learning with Language Models](https://mlsanity.com/paper/2406.04344/) (2024)
- [StraGo: Harnessing Strategic Guidance for Prompt Optimization](https://mlsanity.com/paper/2410.08601/) (2024)
- [A LLM-Powered Automatic Grading Framework with Human-Level Guidelines Optimization](https://mlsanity.com/paper/2410.02165/) (2024)
- [Can Textual Gradient Work in Federated Learning?](https://mlsanity.com/paper/2502.19980/) (2025)
- [An Optimizable Suffix Is Worth A Thousand Templates: Efficient Black-box Jailbreaking without Affirmative Phrases via LLM as Optimizer](https://mlsanity.com/paper/2408.11313/) (2024)
- [AMPO: Automatic Multi-Branched Prompt Optimization](https://mlsanity.com/paper/2410.08696/) (2024)
- [POT: Inducing Overthinking in LLMs via Black-Box Iterative Optimization](https://mlsanity.com/paper/2508.19277/) (2025)
- [SCULPT: Systematic Tuning of Long Prompts](https://mlsanity.com/paper/2410.20788/) (2024)
- [DLPO: Towards a Robust, Efficient, and Generalizable Prompt Optimization Framework from a Deep-Learning Perspective](https://mlsanity.com/paper/2503.13413/) (2025)
- [Metaheuristics and Large Language Models Join Forces: Toward an Integrated Optimization Approach](https://mlsanity.com/paper/2405.18272/) (2024)
- [GRAD-SUM: Leveraging Gradient Summarization for Optimal Prompt Engineering](https://mlsanity.com/paper/2407.12865/) (2024)
- [Optimizing LLM-Based Multi-Agent System with Textual Feedback: A Case Study on Software Development](https://mlsanity.com/paper/2505.16086/) (2025)
- [From Standalone LLMs to Integrated Intelligence: A Survey of Compound Al Systems](https://mlsanity.com/paper/2506.04565/) (2025)
- [Same Question, Different Words: A Latent Adversarial Framework for Prompt Robustness](https://mlsanity.com/paper/2503.01345/) (2025)
- [Tuning-Free Personalized Alignment via Trial-Error-Explain In-Context Learning](https://mlsanity.com/paper/2502.08972/) (2025)
- [Verbalized Graph Representation Learning: A Fully Interpretable Graph Model Based on Large Language Models Throughout the Entire Process](https://mlsanity.com/paper/2410.01457/) (2024)
- [One2set + Large Language Model: Best Partners for Keyphrase Generation](https://mlsanity.com/paper/2410.03421/) (2024)

## Abstract (arXiv)

LLM-based Automatic Prompt Optimization, which typically utilizes LLMs as Prompt Optimizers to self-reflect and refine prompts, has shown promising performance in recent studies. Despite the success, the underlying mechanism of this approach remains unexplored, and the true effectiveness of LLMs as Prompt Optimizers requires further validation. In this work, we conducted a comprehensive study to uncover the actual mechanism of LLM-based Prompt Optimization. Our findings reveal that the LLM optimizers struggle to identify the true causes of errors during reflection, tending to be biased by their own prior knowledge rather than genuinely reflecting on the errors. Furthermore, even when the reflection is semantically valid, the LLM optimizers often fail to generate appropriate prompts for the target models with a single prompt refinement step, partly due to the unpredictable behaviors of the target models. Based on the observations, we introduce a new "Automatic Behavior Optimization" paradigm, which directly optimizes the target model's behavior in a more controllable manner. We hope our study can inspire new directions for automatic prompt optimization development.