mlsanityData to 7 Oct 2026

A Study on the Calibration of In-context Learning

arXiv 2312.04021 ↗First posted 7 Dec 2023cs.CL cs.AI cs.LGcited by 21

Not on the map. Why?

Abstract (arXiv)

Accurate uncertainty quantification is crucial for the safe deployment of machine learning models, and prior research has demonstrated improvements in the calibration of modern language models (LMs). We study in-context learning (ICL), a prevalent method for adapting static LMs through tailored prompts, and examine the balance between performance and calibration across a broad spectrum of natural language understanding and reasoning tasks. Through comprehensive experiments, we observe that, with an increasing number of ICL examples, models initially exhibit increased miscalibration before achieving better calibration and miscalibration tends to arise in low-shot settings. Moreover, we find that methods aimed at improving usability, such as fine-tuning and chain-of-thought (CoT) prompting, can lead to miscalibration and unreliable natural language explanations. Furthermore, we explore recalibration techniques and find that a scaling-binning calibrator can reduce calibration errors consistently.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 46 of its 71 references that are in the map. First posted on arXiv 7 Dec 2023.

Builds on

46 of 71 references.

References · 46

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2017On Calibration of Modern Neural Networkscited by 4,860
  6. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  9. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  10. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  11. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  12. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
34 more
  1. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  2. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  3. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  4. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  5. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  6. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  7. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  8. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  9. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
  10. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  11. 2019Measuring Calibration in Deep Learningcited by 443
  12. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  13. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 402
  14. 2020Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performancecited by 396
  15. 2019Verified Uncertainty Calibrationcited by 307
  16. 2020Calibration of Pre-trained Transformerscited by 307
  17. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  18. 2020Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Makingcited by 281
  19. 2020Selective Question Answering under Domain Shiftcited by 228
  20. 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 174
  21. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 149
  22. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 138
  23. 2020Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertaintycited by 128
  24. 2023Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineeringcited by 91
  25. 2023Mitigating Label Biases for In-context Learningcited by 88
  26. 2021Consistent Accelerated Inference via Confident Adaptive Transformerscited by 73
  27. 2022Stop Measuring Calibration When Humans Disagreecited by 70
  28. 2023Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?cited by 62
  29. 2018WorldTree: A Corpus of Explanation Graphs for Elementary Science Questions supporting Multi-Hop Inferencecited by 60
  30. 2022Prototypical Calibration for Few-shot Learning of Language Modelscited by 58
  31. 2023Large Language Models Help Humans Verify Truthfulness -- Except When They Are Convincingly Wrongcited by 51
  32. 2023When Does Optimizing a Proper Loss Yield Calibration?cited by 35
  33. 2019Calibration, Entropy Rates, and Memory in Language Modelscited by 32
  34. 2023Simple Token-Level Confidence Improves Caption Correctnesscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 8; 2025: 8; 2026: 4.

21 papers.

Sort:

By area:

  1. 2024Uncertainty Estimation and Quantification for LLMs: A Simple Supervised Approachcited by 73
  2. 2024Calibrating Language Models with Adaptive Temperature Scalingcited by 57
  3. 2024Why Larger Language Models Do In-context Learning Differently?cited by 44
  4. 2024Think Twice Before Trusting: Self-Detection for Large Language Models through Comprehensive Answer Reflectioncited by 40
  5. 2025Mind the Confidence Gap: Overconfidence, Calibration, and Distractor Effects in Large Language Modelscited by 39
  6. 2024Evaluating language models as risk scorescited by 24
  7. 2024Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descentcited by 21
  8. 2024Understanding the Effects of Iterative Prompting on Truthfulnesscited by 14
  9. 2023Beyond Output Matching: Bidirectional Alignment for Enhanced In-Context Learningcited by 5
  10. 2025Using Large Language Models to Categorize Strategic Situations and Decipher Motivations Behind Human Behaviorscited by 5
  11. 2025Logits are All We Need to Adapt Closed Modelscited by 4
  12. 2026LLM Performance Predictors: Learning When to Escalate in Hybrid Human-AI Moderation Systemscited by 4
9 more
  1. 2025In-Context Learning (and Unlearning) of Length Biasescited by 3
  2. 2026Frontier Models Can Take Actions at Low Probabilitiescited by 3
  3. 2025Fine-Tuning and Prompt Engineering of LLMs, for the Creation of Multi-Agent AI for Addressing Sustainable Protein Production Challengescited by 2
  4. 2025IA2: Alignment with ICL Activations Improves Supervised Fine-Tuningcited by 1
  5. 2025Music Recommendation with Large Language Models: Challenges, Opportunities, and Evaluationcited by 1
  6. 2026Confidence Should Be Calibrated More Than One Turn Deepcited by 1
  7. 2024Calibrate to Discriminate: Improve In-Context Learning with Label-Free Comparative Inferencecited by 0
  8. 2025COM-BOM: Bayesian Exemplar Search for Efficiently Exploring the Accuracy-Calibration Pareto Frontiercited by 0
  9. 2026A Bayesian Perspective on the Role of Epistemic Uncertainty for Delayed Generalization in In-Context Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Test-time Calibration Learning for Large Language Model Reasoning
  2. 2025Thinking with Nothinking Calibration: A New In-Context Learning Paradigm in Reasoning Large Language Models
  3. 2024Concept-aware Data Construction Improves In-context Learning of Language Models
  4. 2026From Entropy to Calibrated Uncertainty: Training Language Models to Reason About Uncertainty
  5. 2024Bayesian scaling laws for in-context learning
  6. 2025Revisiting Uncertainty Estimation and Calibration of Large Language Models
  7. 2026Probabilistic Calibration Is a Trainable Capability in Language Models
  8. 2023Concept-aware Training Improves In-context Learning Ability of Language Models