mlsanityData to 7 Oct 2026

Large Language Models Are Human-Level Prompt Engineers

arXiv 2211.01910 ↗First posted 3 Nov 2022cs.LG cs.AI cs.CLcited by 1,020

Not on the map. Why?

Cited by 167 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

By conditioning on natural language instructions, large language models (LLMs) have displayed impressive capabilities as general-purpose computers. However, task performance depends significantly on the quality of the prompt used to steer the model, and most effective prompts have been handcrafted by humans. Inspired by classical program synthesis and the human approach to prompt engineering, we propose Automatic Prompt Engineer (APE) for automatic instruction generation and selection. In our method, we treat the instruction as the "program," optimized by searching over a pool of instruction candidates proposed by an LLM in order to maximize a chosen score function. To evaluate the quality of the selected instruction, we evaluate the zero-shot performance of another LLM following the selected instruction. Experiments on 24 NLP tasks show that our automatically generated instructions outperform the prior LLM baseline by a large margin and achieve better or comparable performance to the instructions generated by human annotators on 19/24 tasks. We conduct extensive qualitative and quantitative analyses to explore the performance of APE. We show that APE-engineered prompts can be applied to steer models toward truthfulness and/or informativeness, as well as to improve few-shot learning performance by simply prepending them to standard in-context learning prompts. Please check out our webpage at https://sites.google.com/view/automatic-prompt-engineer.

Key facts. Cited by 1,020 papers in the map (data through 7 Oct 2026). Builds on 49 of its 62 references that are in the map. First posted on arXiv 3 Nov 2022.

Builds on

49 of 62 references.

References · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  9. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2020Scaling Laws for Neural Language Modelscited by 6,411
  12. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
37 more
  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  3. 2021Program Synthesis with Large Language Modelscited by 3,509
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  6. 2022Emergent Abilities of Large Language Modelscited by 2,483
  7. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  10. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  11. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  12. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  13. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  14. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  15. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  16. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  17. 2019How Can We Know What Language Models Know?cited by 953
  18. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  19. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  20. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  21. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  22. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  23. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  24. 2021GPT Understands, Toocited by 715
  25. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  26. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  27. 2016Solving General Arithmetic Word Problemscited by 353
  28. 2021True Few-Shot Learning with Language Modelscited by 343
  29. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  30. 2022Efficient Training of Language Models to Fill in the Middlecited by 248
  31. 2019Commonsense Knowledge Mining from Pretrained Modelscited by 189
  32. 2022Instruction Induction: From Few Examples to Natural Language Task Descriptionscited by 152
  33. 2021Jigsaw: Large Language Models meet Program Synthesiscited by 103
  34. 2021Leveraging Language to Learn Program Abstractions and Search Heuristicscited by 44
  35. 2014Firefly Monte Carlo: Exact MCMC with Subsets of Datacited by 23
  36. 2017Neural Program Meta-Inductioncited by 23
  37. 2021Thinking Aloud: Dynamic Context Generation Improves Zero-Shot Reasoning Performance of GPT-2cited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 221; 2024: 284; 2025: 239; 2026: 271.

1,020 papers.

Sort:

By area:

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  3. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  4. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  5. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  6. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  7. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  8. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  9. 2023Large Language Models as Optimizerscited by 830
  10. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  11. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  12. 2022A Survey on In-context Learningcited by 717
38 more
  1. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  2. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
  3. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 612
  4. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 521
  5. 2024Large Language Models: A Surveycited by 506
  6. 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applicationscited by 428
  7. 2023Cognitive Architectures for Language Agentscited by 422
  8. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  9. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causalitycited by 348
  10. 2024Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programscited by 270
  11. 2024ReEvo: Large Language Models as Hyper-Heuristics with Reflective Evolutioncited by 266
  12. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 257
  13. 2024TextGrad: Automatic "Differentiation" via Textcited by 250
  14. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
  15. 2023PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimizationcited by 226
  16. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 222
  17. 2024Chain-of-Thought Reasoning Without Promptingcited by 208
  18. 2025Video models are zero-shot learners and reasonerscited by 208
  19. 2022Optimizing Prompts for Text-to-Image Generationcited by 204
  20. 2023EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizerscited by 189
  21. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 189
  22. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 187
  23. 2023Efficient Large Language Models: A Surveycited by 183
  24. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  25. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  26. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 176
  27. 2023Talk like a Graph: Encoding Graphs for Large Language Modelscited by 175
  28. 2023Art or Artifice? Large Language Models and the False Promise of Creativitycited by 168
  29. 2023Implicit Chain of Thought Reasoning via Knowledge Distillationcited by 165
  30. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMscited by 160
  31. 2023In-Context Impersonation Reveals Large Language Models' Strengths and Biasescited by 154
  32. 2024Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Modelscited by 150
  33. 2023Large Language Models Understand and Can be Enhanced by Emotional Stimulicited by 143
  34. 2023Towards Making the Most of ChatGPT for Machine Translationcited by 138
  35. 2023EvoPrompting: Language Models for Code-Level Neural Architecture Searchcited by 129
  36. 2023Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generationcited by 128
  37. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  38. 2023Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselvescited by 123

50 most cited of 1,020 All 1,020 (JSON)