mlsanityData to 14 Oct 2026

A Systematic Survey of Automatic Prompt Optimization Techniques

arXiv 2502.16923 ↗First posted 24 Feb 2025cs.CL cs.AIcited by 63

Not on the map. Why?

Abstract (arXiv)

Since the advent of large language models (LLMs), prompt engineering has been a crucial step for eliciting desired responses for various Natural Language Processing (NLP) tasks. However, prompt engineering remains an impediment for end users due to rapid advances in models, tasks, and associated best practices. To mitigate this, Automatic Prompt Optimization (APO) techniques have recently emerged that use various automated techniques to help improve the performance of LLMs on various tasks. In this paper, we present a comprehensive survey summarizing the current progress and remaining challenges in this field. We provide a formal definition of APO, a 5-part unifying framework, and then proceed to rigorously categorize all relevant works based on their salient features therein. We hope to spur further research guided by our framework.

Key facts. Cited by 63 papers in the map (data through 14 Oct 2026). Builds on 59 of its 103 references that are in the map. First posted on arXiv 24 Feb 2025.

Builds on

59 of 103 references.

References · 59

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  5. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  10. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  11. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  12. 2015A large annotated corpus for learning natural language inferencecited by 1,847
38 more
  1. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  2. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  3. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  4. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  5. 2018Neural Network Acceptability Judgmentscited by 958
  6. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  7. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  8. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 743
  9. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  10. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 650
  11. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 551
  12. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 535
  13. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 366
  14. 2017"Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detectioncited by 363
  15. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  16. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 327
  17. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 324
  18. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 253
  19. 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 236
  20. 2020HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verificationcited by 201
  21. 2022Instruction Induction: From Few Examples to Natural Language Task Descriptionscited by 152
  22. 2020Open-Domain Question Answering Goes Conversational via Question Rewritingcited by 148
  23. 2023Large Language Models Understand and Can be Enhanced by Emotional Stimulicited by 146
  24. 2023Large Language Models as Evolutionary Optimizerscited by 124
  25. 2024Direct Preference Optimization with an Offsetcited by 122
  26. 2023Guiding Large Language Models via Directional Stimulus Promptingcited by 95
  27. 2024Reinforcement Learning for LLM Post-Training: A Surveycited by 92
  28. 2024Improving Text-to-Image Consistency via Automatic Prompt Optimizationcited by 90
  29. 2023Reprompting: Automated Chain-of-Thought Prompt Inference Through Gibbs Samplingcited by 47
  30. 2024PRewrite: Prompt Rewriting with Reinforcement Learningcited by 45
  31. 2023What's the Magic Word? A Control Theory of LLM Promptingcited by 41
  32. 2020A Dataset and Baselines for Visual Question Answering on Artcited by 38
  33. 2023Query-Dependent Prompt Evaluation and Optimization with Offline Inverse RLcited by 37
  34. 2022PromptBoosting: Black-Box Text Classification with Ten Forward Passescited by 36
  35. 2021DOCKSTRING: easy molecular docking yields better benchmarks for ligand designcited by 35
  36. 2024Symbolic Prompt Program Search: A Structure-Aware Approach to Efficient Compile-Time Prompt Optimizationcited by 30
  37. 2024SPRIG: Improving Large Language Model Performance by System Prompt Optimizationcited by 29
  38. 2023PREFER: Prompt Ensemble Learning via Feedback-Reflect-Refinecited by 28

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 21; 2026: 42.

63 papers.

Sort:

By area:

  1. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 170
  2. 2026DeltaEvolve: Accelerating Scientific Discovery through Momentum-Driven Evolutioncited by 26
  3. 2026EvoTool: Self-Evolving Tool-Use Policy Optimization in LLM Agents via Blame-Aware Mutation and Diversity-Aware Selectioncited by 24
  4. 2026CooperBench: Why Coding Agents Cannot be Your Teammates Yetcited by 23
  5. 2026HarnessCompass: Guiding Automatic Harness Evolution toward Generalizable and Effective Agent Harnessescited by 17
  6. 2025Prompt-MII: Meta-Learning Instruction Induction for LLMscited by 9
  7. 2025LLM Prompt Duel Optimizer: Efficient Label-Free Prompt Optimizationcited by 8
  8. 2026Managing Procedural Memory in LLM Agents: Control, Adaptation, and Evaluationcited by 8
  9. 2025No Loss, No Gain: Gated Refinement and Adaptive Compression for Prompt Optimizationcited by 7
  10. 2025Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for MLLMscited by 7
  11. 2025Automatic Prompt Optimization for Knowledge Graph Construction: Insights from an Empirical Studycited by 6
  12. 2025ProCut: LLM Prompt Compression via Attribution Estimationcited by 6
38 more
  1. 2025GenPilot: A Multi-Agent System for Test-Time Prompt Optimization in Image Generationcited by 6
  2. 2025POPI: Personalizing LLMs via Optimized Natural Language Preference Inferencecited by 6
  3. 2025COMMUNITYNOTES: A Dataset for Exploring the Helpfulness of Fact-Checking Explanationscited by 5
  4. 2026MEMO: Memory-Augmented Model Context Optimization for Robust Multi-Turn Multi-Agent LLM Gamescited by 5
  5. 2026MAS-PromptBench: When Does Prompt Optimization Improve Multi-Agent LLM Systems?cited by 5
  6. 2025Beyond One-Size-Fits-All: Inversion Learning for Highly Effective NLG Evaluation Promptscited by 4
  7. 2025Talk Less, Call Right: Enhancing Role-Play LLM Agents with Automatic Prompt Optimization and Role Promptingcited by 4
  8. 2025How to Auto-optimize Prompts for Domain Tasks? Adaptive Prompting and Reasoning through Evolutionary Domain Knowledge Adaptationcited by 4
  9. 2026Rubrics as an Attack Surface: Stealthy Preference Drift in LLM Judgescited by 4
  10. 2025Transduction is All You Need for Structured Data Workflowscited by 3
  11. 2026JudgeFlow: Agentic Workflow Optimization via Block Judgecited by 3
  12. 2026Prompt-Level Distillation: A Non-Parametric Alternative to Model Fine-Tuning for Efficient Reasoningcited by 3
  13. 2026Experience as a Compass: Multi-agent RAG with Evolving Orchestration and Agent Promptscited by 3
  14. 2026AgentGrad: Intervention-guided Prompt Optimization for Multi Agent Systemscited by 3
  15. 2025Something Just Like TRuST : Toxicity Recognition of Span and Targetcited by 2
  16. 2025Through the Judge's Eyes: Inferred Thinking Traces Improve Reliability of LLM Raterscited by 2
  17. 2025Textual Gradients are a Flawed Metaphor for Automatic Prompt Optimizationcited by 2
  18. 2026MAC: Multi-Agent Constitution Learningcited by 2
  19. 2026DetPO: In-Context Learning with Multi-Modal LLMs for Few-Shot Object Detectioncited by 2
  20. 2026LPDS: Evaluating LLM Robustness Through Logic-Preserving Difficulty Scalingcited by 2
  21. 2025SynthEHR-Eviction: Enhancing Eviction SDoH Detection with LLM-Augmented Synthetic EHR Datacited by 1
  22. 2025GreenTEA: Gradient Descent with Topic-modeling and Evolutionary Auto-promptingcited by 1
  23. 2025UniAPO: Unified Multimodal Automated Prompt Optimizationcited by 1
  24. 2026ClinicalReTrial: Clinical Trial Redesign with Self-Evolving Agentscited by 1
  25. 2026POaaS: Minimal-Edit Prompt Optimization as a Service to Lift Accuracy and Cut Hallucinations on On-Device sLLMscited by 1
  26. 2026How Confident Is the First Token? An Uncertainty-Calibrated Prompt Optimization Framework for Large Language Model Classification and Understandingcited by 1
  27. 2026Exploiting LLM-as-a-Judge Disposition on Free Text Legal QA via Prompt Optimizationcited by 1
  28. 2026Embedding by Elicitation: Dynamic Representations for Bayesian Optimization of System Promptscited by 1
  29. 2026Reflective Prompt Tuning through Language Model Function-Callingcited by 1
  30. 2026How Far Can Prompting Go for Minimal-Edit Ukrainian Grammatical Error Correction?cited by 1
  31. 20252K-Characters-10K-Stories: A Quality-Gated Stylized Narrative Dataset with Disentangled Control and Sequence Consistencycited by 0
  32. 2026Responsible AI for General-Purpose Systems: Overview, Challenges, and A Path Forwardcited by 0
  33. 2026Towards Efficient and Robust Linguistic Emotion Diagnosis for Mental Health via Multi-Agent Instruction Refinementcited by 0
  34. 2026GFlowPO: Generative Flow Network as a Language Model Prompt Optimizercited by 0
  35. 2026ADAPT: Hybrid Prompt Optimization for LLM Feature Visualizationcited by 0
  36. 2026CRAFT: Cost-aware Refinement And Front-aware Tuning of Promptscited by 0
  37. 2026SAGE: Stochastic Prompt Optimization via Agent-Guided Explorationcited by 0
  38. 2026Towards Spec Learning: Inference-Time Alignment from Preference Pairscited by 0

50 most cited of 63 All 63 (JSON)