mlsanityData to 14 Oct 2026

Prompt Engineering a Prompt Engineer

arXiv 2311.05661 ↗First posted 9 Nov 2023cs.CL cs.AI cs.LGcited by 53

Not on the map. Why?

Abstract (arXiv)

Prompt engineering is a challenging yet crucial task for optimizing the performance of large language models on customized tasks. It requires complex reasoning to examine the model's errors, hypothesize what is missing or misleading in the current prompt, and communicate the task with clarity. While recent works indicate that large language models can be meta-prompted to perform automatic prompt engineering, we argue that their potential is limited due to insufficient guidance for complex reasoning in the meta-prompt. We fill this gap by infusing into the meta-prompt three key components: detailed descriptions, context specification, and a step-by-step reasoning template. The resulting method, named PE2, exhibits remarkable versatility across diverse language tasks. It finds prompts that outperform "let's think step by step" by 6.3% on MultiArith and 3.1% on GSM8K, and outperforms competitive baselines on counterfactual tasks by 6.9%. Further, we show that PE2 can make targeted and highly specific prompt edits, rectify erroneous prompts, and induce multi-step plans for complex tasks.

Key facts. Cited by 53 papers in the map (data through 14 Oct 2026). Builds on 37 of its 60 references that are in the map. First posted on arXiv 9 Nov 2023.

Builds on

37 of 60 references.

References · 37

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  6. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  7. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  8. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  9. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  10. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  11. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  12. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
25 more
  1. 2023Teaching Large Language Models to Self-Debugcited by 988
  2. 2019How Can We Know What Language Models Know?cited by 959
  3. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  4. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  5. 2022Large Language Models Can Self-Improvecited by 664
  6. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 653
  7. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 606
  8. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 551
  9. 2023Language Models can Solve Computer Taskscited by 372
  10. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 366
  11. 2016Solving General Arithmetic Word Problemscited by 360
  12. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 246
  13. 2023PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimizationcited by 233
  14. 2023REFINER: Reasoning Feedback on Intermediate Representationscited by 205
  15. 2022Instruction Induction: From Few Examples to Natural Language Task Descriptionscited by 152
  16. 2023Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generationcited by 148
  17. 2023The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuningcited by 108
  18. 2023Controlled Text Generation with Natural Language Instructionscited by 87
  19. 2023Evaluating Large Language Models on Controlled Generation Taskscited by 73
  20. 2020Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselvescited by 58
  21. 2023Better Zero-Shot Reasoning with Self-Adaptive Promptingcited by 52
  22. 2022A Modern Self-Referential Weight Matrix That Learns to Modify Itselfcited by 41
  23. 2023Robust Prompt Optimization for Large Language Models Against Distribution Shiftscited by 19
  24. 2022Controllable Text Generation with Language Constraintscited by 18
  25. 2023Evoke: Evoking Critical Thinking Abilities in LLMs via Reviewer-Author Prompt Editingcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 14; 2025: 25; 2026: 14.

53 papers.

Sort:

By area:

  1. 2024TextGrad: Automatic "Differentiation" via Textcited by 273
  2. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  3. 2024Efficient multi-prompt evaluation of LLMscited by 73
  4. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 71
  5. 2024Teach Better or Show Smarter? On Instructions and Exemplars in Automatic Prompt Optimizationcited by 29
  6. 2024Aviary: training language agents on challenging scientific taskscited by 26
  7. 2024LogiCode: an LLM-Driven Framework for Logical Anomaly Detectioncited by 19
  8. 2025MARS: Multi-Agent Adaptive Reasoning with Socratic Guidance for Automated Prompt Optimizationcited by 15
  9. 2024From Prompt Engineering to Prompt Science With Human in the Loopcited by 12
  10. 2024GReaTer: Gradients over Reasoning Makes Smaller Language Models Strong Prompt Optimizerscited by 12
  11. 2024Learning to Reason via Program Generation, Emulation, and Searchcited by 10
  12. 2025Optimizing LLM-Based Multi-Agent System with Textual Feedback: A Case Study on Software Developmentcited by 7
38 more
  1. 2025Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for MLLMscited by 7
  2. 2024Leveraging Large Language Models to Detect npm Malicious Packagescited by 6
  3. 2025Large Language Models and Evolutionary Computation: A Critical Review of Bidirectional Interaction, Automated Algorithm Design, and Co-Adaptive Systemscited by 6
  4. 2025metaTextGrad: Automatically optimizing language model optimizerscited by 6
  5. 2026Test-Driven AI Agent Definition (TDAD): Compiling Tool-Using Agents from Behavioral Specificationscited by 6
  6. 2024Navigating the Landscape of Large Language Models: A Comprehensive Review and Analysis of Paradigms and Fine-Tuning Strategiescited by 5
  7. 2024Exploring Prompt Engineering: A Systematic Review with SWOT Analysiscited by 5
  8. 2025Multimodal-to-Text Prompt Engineering in Large Language Models Using Feature Embeddings for GNSS Interference Characterizationcited by 4
  9. 2026Brevity Constraints Reverse Performance Hierarchies in Language Modelscited by 4
  10. 2025TAPO: Task-Referenced Adaptation for Prompt Optimizationcited by 3
  11. 2024Renaissance of Literate Programming in the Era of LLMs: Enhancing LLM-Based Code Generation in Large-Scale Projectscited by 3
  12. 2025Transduction is All You Need for Structured Data Workflowscited by 3
  13. 2025GPS: General Per-Sample Promptercited by 3
  14. 2026AutoFocus: Uncertainty-Aware Active Visual Search for GUI Groundingcited by 3
  15. 2025Function Induction and Task Generalization: An Interpretability Study with Off-by-One Additioncited by 2
  16. 2025Retrieval augmented generation based dynamic prompting for few-shot biomedical named entity recognition using large language modelscited by 2
  17. 2025Large Language Models for Oral History Understanding with Text Classification and Sentiment Analysiscited by 2
  18. 2025PIAST: Rapid Prompting with In-context Augmentation for Scarce Training datacited by 2
  19. 2024Open Artificial Knowledgecited by 1
  20. 2025Knowledge Acquisition on Mass-shooting Events via LLMs for AI-Driven Justicecited by 1
  21. 2025Understanding Prompt Programming Tasks and Questionscited by 1
  22. 2025Are You There God? Lightweight Narrative Annotation of Christian Fiction with LMscited by 1
  23. 2025Beyond Elicitation: Provision-based Prompt Optimization for Knowledge-Intensive Taskscited by 1
  24. 2026A Training-Free Guess What Vision Language Model from Snippets to Open-Vocabulary Object Detectioncited by 1
  25. 2026Decoding ML Decision: An Agentic Reasoning Framework for Large-Scale Ranking Systemcited by 1
  26. 2026Helix: A Dual-Helix Co-Evolutionary Multi-Agent System for Prompt Optimization and Question Reformulationcited by 1
  27. 2025A Multi-Agent Probabilistic Inference Framework Inspired by Kairanban-Style CoT System with IdoBata Conversation for Debiasingcited by 0
  28. 2025A Language-Signal-Vision Multimodal Framework for Multitask Cardiac Analysiscited by 0
  29. 2025Model-Driven Quantum Code Generation Using Large Language Models and Retrieval-Augmented Generationcited by 0
  30. 2025Prompt reinforcing for long-term planning of large language modelscited by 0
  31. 2025ELPO: Ensemble Learning Based Prompt Optimization for Large Language Modelscited by 0
  32. 2025Prompt-Based Value Steering of Large Language Modelscited by 0
  33. 2025Transforming Monolithic Foundation Models into Embodied Multi-Agent Architectures for Human-Robot Collaborationcited by 0
  34. 2026Towards Efficient and Robust Linguistic Emotion Diagnosis for Mental Health via Multi-Agent Instruction Refinementcited by 0
  35. 2026PromptHelper: A Prompt Recommender System for Encouraging Creativity in AI Chatbot Interactionscited by 0
  36. 2026GFlowPO: Generative Flow Network as a Language Model Prompt Optimizercited by 0
  37. 2026Evaluating Prompt Engineering Techniques for RAG in Small Language Models: A Multi-Hop QA Approachcited by 0
  38. 2026Position: Vector Prompt Interfaces Should Be Exposed to Enable Customization of Large Language Modelscited by 0

50 most cited of 53 All 53 (JSON)