mlsanityData to 7 Oct 2026

Flatness-Aware Prompt Selection Improves Accuracy and Sample Efficiency

arXiv 2305.10713 ↗First posted 18 May 2023cs.CL cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

With growing capabilities of large language models, prompting them has become the dominant way to access them. This has motivated the development of strategies for automatically selecting effective language prompts. In this paper, we introduce prompt flatness, a new metric to quantify the expected utility of a language prompt. This metric is inspired by flatness regularization in statistical learning that quantifies the robustness of the model towards its parameter perturbations. We provide theoretical foundations for this metric and its relationship with other prompt selection metrics, providing a comprehensive understanding of existing methods. Empirically, we show that combining prompt flatness with existing metrics improves both performance and sample efficiency. Our metric outperforms the previous prompt selection metrics with an average increase of 5% in accuracy and 10% in Pearson correlation across 6 classification benchmarks.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 40 of its 62 references that are in the map. First posted on arXiv 18 May 2023.

Builds on

40 of 62 references.

References · 40

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  8. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  9. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  10. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  11. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  12. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
28 more
  1. 2019How Can We Know What Language Models Know?cited by 953
  2. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  3. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 737
  4. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  5. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
  6. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
  7. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  8. 2021True Few-Shot Learning with Language Modelscited by 343
  9. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  10. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  11. 2021Reframing Instructional Prompts to GPTk's Languagecited by 176
  12. 2022Towards Understanding Sharpness-Aware Minimizationcited by 145
  13. 2021An Empirical Investigation of the Role of Pre-training in Lifelong Learningcited by 134
  14. 2022GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Modelscited by 127
  15. 2022Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learningcited by 122
  16. 2021Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learnerscited by 115
  17. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  18. 2020Regularizing Neural Networks via Adversarial Model Perturbationcited by 78
  19. 2022On the Relation between Sensitivity and Accuracy in In-context Learningcited by 71
  20. 2022HyperPrompt: Prompt-based Task-Conditioning of Transformerscited by 67
  21. 2021Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Promptscited by 64
  22. 2021Relating Adversarially Robust Generalization to Flat Minimacited by 57
  23. 2022TEMPERA: Test-Time Prompting via Reinforcement Learningcited by 50
  24. 2019Shaping the learning landscape in neural networks around wide flat minimacited by 47
  25. 2022Co-training Improves Prompt-based Learning for Large Language Modelscited by 24
  26. 2022Train Flat, Then Compress: Sharpness-Aware Minimization Learns More Compressible Modelscited by 20
  27. 2022On Robust Prefix-Tuning for Text Classificationcited by 15
  28. 2022SQuAT: Sharpness- and Quantization-Aware Training for BERTcited by 7

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 2.

5 papers.

  1. 2023Unleashing the potential of prompt engineering for large language modelscited by 72
  2. 2023Do pretrained Transformers Learn In-Context by Gradient Descent?cited by 18
  3. 2024Enhancing Domain Adaptation through Prompt Gradient Alignmentcited by 8
  4. 2024Hierarchical Prompting Taxonomy: A Universal Evaluation Framework for Large Language Models Aligned with Human Cognitive Principlescited by 4
  5. 2023Gradient constrained sharpness-aware prompt learning for vision-language modelscited by 1