mlsanityData to 7 Oct 2026

Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity

arXiv 2104.08786 ↗First posted 18 Apr 2021cs.CL cs.AIcited by 1,122

Not on the map. Why?

Cited by 94 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

When primed with only a handful of training samples, very large, pretrained language models such as GPT-3 have shown competitive results when compared to fully-supervised, fine-tuned, large, pretrained language models. We demonstrate that the order in which the samples are provided can make the difference between near state-of-the-art and random guess performance: essentially some permutations are "fantastic" and some not. We analyse this phenomenon in detail, establishing that: it is present across model sizes (even for the largest current models), it is not related to a specific subset of samples, and that a given good permutation for one model is not transferable to another. While one could use a development set to determine which permutations are performant, this would deviate from the true few-shot setting as it requires additional annotated data. Instead, we use the generative nature of language models to construct an artificial development set and based on entropy statistics of the candidate permutations on this set, we identify performant prompts. Our method yields a 13% relative improvement for GPT-family models across eleven different established text classification tasks.

Key facts. Cited by 1,122 papers in the map (data through 7 Oct 2026). Builds on 17 of its 33 references that are in the map. First posted on arXiv 18 Apr 2021.

Builds on

17 of 33 references.

References · 17

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  6. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  7. 2018Deep contextualized word representationscited by 2,285
  8. 2019Language Models as Knowledge Bases?cited by 1,922
  9. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  10. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  11. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  12. 2019How Can We Know What Language Models Know?cited by 953
5 more
  1. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  2. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
  3. 2019Commonsense Knowledge Mining from Pretrained Modelscited by 189
  4. 2020How Context Affects Language Models' Factual Predictionscited by 181
  5. 2015Ask Me Anything: Dynamic Memory Networks for Natural Language Processingcited by 116

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 18; 2022: 107; 2023: 300; 2024: 329; 2025: 189; 2026: 179.

1,122 papers.

Sort:

By area:

  1. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  2. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2023A Survey of Large Language Modelscited by 2,537
  5. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  6. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  7. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  8. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  9. 2023A Survey on Multimodal Large Language Modelscited by 898
  10. 2023Large Language Models as Optimizerscited by 830
  11. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  12. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
38 more
  1. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  2. 2022A Survey on In-context Learningcited by 717
  3. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  4. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  5. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
  6. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 521
  7. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 506
  8. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  9. 2021MetaICL: Learning to Learn In Contextcited by 450
  10. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  11. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  12. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  13. 2023Augmented Language Models: a Surveycited by 398
  14. 2022Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)cited by 396
  15. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  16. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
  17. 2023A Survey on Large Language Models for Recommendationcited by 378
  18. 2022DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Modelscited by 376
  19. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  20. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  21. 2021True Few-Shot Learning with Language Modelscited by 343
  22. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  23. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thoughtcited by 334
  24. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 333
  25. 2023Large Language Models are Zero-Shot Rankers for Recommender Systemscited by 307
  26. 2023Large Language Models are Effective Text Rankers with Pairwise Ranking Promptingcited by 302
  27. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
  28. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 295
  29. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  30. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  31. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  32. 2023Can We Edit Factual Knowledge by In-Context Learning?cited by 284
  33. 2021AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Promptscited by 265
  34. 2023Sentiment Analysis in the Era of Large Language Models: A Reality Checkcited by 262
  35. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  36. 2022Visual Prompting via Image Inpaintingcited by 248
  37. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 242
  38. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 240

50 most cited of 1,122 All 1,122 (JSON)