# Prompt Engineering a Prompt Engineer

arXiv: https://arxiv.org/abs/2311.05661 · HTML: https://mlsanity.com/paper/2311.05661/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 60 references that are in the map.
- First posted on arXiv 9 Nov 2023.

## Builds on (most cited first, 30 of 37)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Show Your Work: Scratchpads for Intermediate Computation with Language Models](https://mlsanity.com/paper/2112.00114/) (2021)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [Automatic Prompt Optimization with "Gradient Descent" and Beam Search](https://mlsanity.com/paper/2305.03495/) (2023)
- [Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm](https://mlsanity.com/paper/2102.07350/) (2021)
- [Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution](https://mlsanity.com/paper/2309.16797/) (2023)
- [Language Models can Solve Computer Tasks](https://mlsanity.com/paper/2303.17491/) (2023)
- [RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning](https://mlsanity.com/paper/2205.12548/) (2022)
- [Solving General Arithmetic Word Problems](https://mlsanity.com/paper/1608.01413/) (2016)
- [Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks](https://mlsanity.com/paper/2307.02477/) (2023)
- [PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization](https://mlsanity.com/paper/2310.16427/) (2023)
- [REFINER: Reasoning Feedback on Intermediate Representations](https://mlsanity.com/paper/2304.01904/) (2023)
- [Instruction Induction: From Few Examples to Natural Language Task Descriptions](https://mlsanity.com/paper/2205.10782/) (2022)
- [Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation](https://mlsanity.com/paper/2310.02304/) (2023)
- [The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning](https://mlsanity.com/paper/2305.14045/) (2023)
- [Controlled Text Generation with Natural Language Instructions](https://mlsanity.com/paper/2304.14293/) (2023)

## Built on by (most cited first, 30 of 53)

- [TextGrad: Automatic "Differentiation" via Text](https://mlsanity.com/paper/2406.07496/) (2024)
- [Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems](https://mlsanity.com/paper/2504.01990/) (2025)
- [Efficient multi-prompt evaluation of LLMs](https://mlsanity.com/paper/2405.17202/) (2024)
- [When LLMs Meet Cybersecurity: A Systematic Literature Review](https://mlsanity.com/paper/2405.03644/) (2024)
- [Teach Better or Show Smarter? On Instructions and Exemplars in Automatic Prompt Optimization](https://mlsanity.com/paper/2406.15708/) (2024)
- [Aviary: training language agents on challenging scientific tasks](https://mlsanity.com/paper/2412.21154/) (2024)
- [LogiCode: an LLM-Driven Framework for Logical Anomaly Detection](https://mlsanity.com/paper/2406.04687/) (2024)
- [MARS: Multi-Agent Adaptive Reasoning with Socratic Guidance for Automated Prompt Optimization](https://mlsanity.com/paper/2503.16874/) (2025)
- [From Prompt Engineering to Prompt Science With Human in the Loop](https://mlsanity.com/paper/2401.04122/) (2024)
- [GReaTer: Gradients over Reasoning Makes Smaller Language Models Strong Prompt Optimizers](https://mlsanity.com/paper/2412.09722/) (2024)
- [Learning to Reason via Program Generation, Emulation, and Search](https://mlsanity.com/paper/2405.16337/) (2024)
- [Optimizing LLM-Based Multi-Agent System with Textual Feedback: A Case Study on Software Development](https://mlsanity.com/paper/2505.16086/) (2025)
- [Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for MLLMs](https://mlsanity.com/paper/2510.09201/) (2025)
- [Leveraging Large Language Models to Detect npm Malicious Packages](https://mlsanity.com/paper/2403.12196/) (2024)
- [Large Language Models and Evolutionary Computation: A Critical Review of Bidirectional Interaction, Automated Algorithm Design, and Co-Adaptive Systems](https://mlsanity.com/paper/2505.15741/) (2025)
- [metaTextGrad: Automatically optimizing language model optimizers](https://mlsanity.com/paper/2505.18524/) (2025)
- [Test-Driven AI Agent Definition (TDAD): Compiling Tool-Using Agents from Behavioral Specifications](https://mlsanity.com/paper/2603.08806/) (2026)
- [Navigating the Landscape of Large Language Models: A Comprehensive Review and Analysis of Paradigms and Fine-Tuning Strategies](https://mlsanity.com/paper/2404.09022/) (2024)
- [Exploring Prompt Engineering: A Systematic Review with SWOT Analysis](https://mlsanity.com/paper/2410.12843/) (2024)
- [Multimodal-to-Text Prompt Engineering in Large Language Models Using Feature Embeddings for GNSS Interference Characterization](https://mlsanity.com/paper/2501.05079/) (2025)
- [Brevity Constraints Reverse Performance Hierarchies in Language Models](https://mlsanity.com/paper/2604.00025/) (2026)
- [TAPO: Task-Referenced Adaptation for Prompt Optimization](https://mlsanity.com/paper/2501.06689/) (2025)
- [Renaissance of Literate Programming in the Era of LLMs: Enhancing LLM-Based Code Generation in Large-Scale Projects](https://mlsanity.com/paper/2502.17441/) (2024)
- [Transduction is All You Need for Structured Data Workflows](https://mlsanity.com/paper/2508.15610/) (2025)
- [GPS: General Per-Sample Prompter](https://mlsanity.com/paper/2511.21714/) (2025)
- [AutoFocus: Uncertainty-Aware Active Visual Search for GUI Grounding](https://mlsanity.com/paper/2605.02630/) (2026)
- [Function Induction and Task Generalization: An Interpretability Study with Off-by-One Addition](https://mlsanity.com/paper/2507.09875/) (2025)
- [Retrieval augmented generation based dynamic prompting for few-shot biomedical named entity recognition using large language models](https://mlsanity.com/paper/2508.06504/) (2025)
- [Large Language Models for Oral History Understanding with Text Classification and Sentiment Analysis](https://mlsanity.com/paper/2508.06729/) (2025)
- [PIAST: Rapid Prompting with In-context Augmentation for Scarce Training data](https://mlsanity.com/paper/2512.11013/) (2025)

## Abstract (arXiv)

Prompt engineering is a challenging yet crucial task for optimizing the performance of large language models on customized tasks. It requires complex reasoning to examine the model's errors, hypothesize what is missing or misleading in the current prompt, and communicate the task with clarity. While recent works indicate that large language models can be meta-prompted to perform automatic prompt engineering, we argue that their potential is limited due to insufficient guidance for complex reasoning in the meta-prompt. We fill this gap by infusing into the meta-prompt three key components: detailed descriptions, context specification, and a step-by-step reasoning template. The resulting method, named PE2, exhibits remarkable versatility across diverse language tasks. It finds prompts that outperform "let's think step by step" by 6.3% on MultiArith and 3.1% on GSM8K, and outperforms competitive baselines on counterfactual tasks by 6.9%. Further, we show that PE2 can make targeted and highly specific prompt edits, rectify erroneous prompts, and induce multi-step plans for complex tasks.