mlsanityData to 7 Oct 2026

Calibrate Before Use: Improving Few-Shot Performance of Language Models

arXiv 2102.09690 ↗First posted 19 Feb 2021cs.CL cs.LGcited by 1,371

Not on the map. Why?

Cited by 101 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

GPT-3 can perform numerous tasks when provided a natural language prompt that contains a few training examples. We show that this type of few-shot learning can be unstable: the choice of prompt format, training examples, and even the order of the training examples can cause accuracy to vary from near chance to near state-of-the-art. We demonstrate that this instability arises from the bias of language models towards predicting certain answers, e.g., those that are placed near the end of the prompt or are common in the pre-training data. To mitigate this, we first estimate the model's bias towards each answer by asking for its prediction when given the training prompt and a content-free test input such as "N/A". We then fit calibration parameters that cause the prediction for this input to be uniform across answers. On a diverse set of tasks, this contextual calibration procedure substantially improves GPT-3 and GPT-2's average accuracy (up to 30.0% absolute) and reduces variance across different choices of the prompt.

Key facts. Cited by 1,371 papers in the map (data through 7 Oct 2026). Builds on 22 of its 35 references that are in the map. First posted on arXiv 19 Feb 2021.

Builds on

22 of 35 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2017On Calibration of Modern Neural Networkscited by 4,860
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  5. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  6. 2019Language Models as Knowledge Bases?cited by 1,922
  7. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  8. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  9. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  10. 2019How Can We Know What Language Models Know?cited by 953
  11. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  12. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
10 more
  1. 2019COMET: Commonsense Transformers for Automatic Knowledge Graph Constructioncited by 450
  2. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
  3. 2020AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Promptscited by 246
  4. 2018A Simple Method for Commonsense Reasoningcited by 223
  5. 2018Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Contextcited by 113
  6. 2019Learning and Evaluating General Linguistic Intelligencecited by 78
  7. 2019Barack's Wife Hillary: Using Knowledge-Graphs for Fact-Aware Language Modelingcited by 71
  8. 2017The Effect of Different Writing Tasks on Linguistic Style: A Case Study of the ROC Story Cloze Taskcited by 39
  9. 2019Studying the Inductive Biases of RNNs with Synthetic Variations of Natural Languagescited by 37
  10. 2019Calibration, Entropy Rates, and Memory in Language Modelscited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 52; 2022: 156; 2023: 368; 2024: 328; 2025: 231; 2026: 236.

1,371 papers.

Sort:

By area:

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2023A Survey of Large Language Modelscited by 2,537
  6. 2022Emergent Abilities of Large Language Modelscited by 2,483
  7. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  9. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  10. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  11. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  12. 2021Ethical and social risks of harm from Language Modelscited by 957
38 more
  1. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  2. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  3. 2023Large Language Models as Optimizerscited by 830
  4. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  5. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  6. 2022A Survey on In-context Learningcited by 717
  7. 2021GPT Understands, Toocited by 715
  8. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  9. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  10. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  11. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  12. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  13. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  14. 2023Segment Everything Everywhere All at Oncecited by 546
  15. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  16. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  17. 2021MetaICL: Learning to Learn In Contextcited by 450
  18. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  19. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  20. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
  21. 2023Larger language models do in-context learning differentlycited by 386
  22. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  23. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  24. 2023Adapting Language Models to Compress Contextscited by 350
  25. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  26. 2021True Few-Shot Learning with Language Modelscited by 343
  27. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  28. 2022Large Language Models Are Reasoning Teacherscited by 318
  29. 2023Large Language Models are Zero-Shot Rankers for Recommender Systemscited by 307
  30. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  31. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  32. 2023Can We Edit Factual Knowledge by In-Context Learning?cited by 284
  33. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  34. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 273
  35. 2022Prompting GPT-3 To Be Reliablecited by 258
  36. 2023Reward Design with Language Modelscited by 257
  37. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  38. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 253

50 most cited of 1,371 All 1,371 (JSON)