mlsanityData to 7 Oct 2026

Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineering

arXiv 2309.17249 ↗First posted 29 Sep 2023cs.CL cs.AI cs.LGcited by 91

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Prompting and in-context learning (ICL) have become efficient learning paradigms for large language models (LLMs). However, LLMs suffer from prompt brittleness and various bias factors in the prompt, including but not limited to the formatting, the choice verbalizers, and the ICL examples. To address this problem that results in unexpected performance degradation, calibration methods have been developed to mitigate the effects of these biases while recovering LLM performance. In this work, we first conduct a systematic analysis of the existing calibration methods, where we both provide a unified view and reveal the failure cases. Inspired by these analyses, we propose Batch Calibration (BC), a simple yet intuitive method that controls the contextual bias from the batched input, unifies various prior approaches, and effectively addresses the aforementioned issues. BC is zero-shot, inference-only, and incurs negligible additional costs. In the few-shot setup, we further extend BC to allow it to learn the contextual bias from labeled data. We validate the effectiveness of BC with PaLM 2-(S, M, L) and CLIP models and demonstrate state-of-the-art performance over previous calibration baselines across more than 10 natural language understanding and image classification tasks.

Key facts. Cited by 91 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 59 of its 73 references that are in the map. First posted on arXiv 29 Sep 2023.

Builds on

59 of 73 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Better Zero-Shot Reasoning with Self-Adaptive Prompting23 May

Other references · 58

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2017On Calibration of Modern Neural Networkscited by 4,860
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  11. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  12. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
38 more
  1. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  2. 2013Describing Textures in the Wildcited by 2,110
  3. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  4. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  5. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  6. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  7. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  8. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  9. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  10. 2023PaLM 2 Technical Reportcited by 1,154
  11. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  12. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  13. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  14. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  15. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  16. 2023Large Language Modelscited by 774
  17. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  18. 2020Improving robustness against common corruptions by covariate shift adaptationcited by 427
  19. 2023Larger language models do in-context learning differentlycited by 386
  20. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  21. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  22. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  23. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249
  24. 2020Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shiftcited by 235
  25. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 163
  26. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 149
  27. 2021The Norm Must Go On: Dynamic Unsupervised Domain Adaptation by Normalizationcited by 131
  28. 2022GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Modelscited by 127
  29. 2023TTN: A Domain-Shift Aware Batch Normalization in Test-Time Adaptationcited by 101
  30. 2022An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labelscited by 88
  31. 2023Mitigating Label Biases for In-context Learningcited by 88
  32. 2022On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Modelcited by 78
  33. 2022Black-box Prompt Learning for Pre-trained Language Modelscited by 70
  34. 2023BlackVIP: Black-Box Visual Prompting for Robust Transfer Learningcited by 70
  35. 2022Prototypical Calibration for Few-shot Learning of Language Modelscited by 58
  36. 2021SITA: Single Image Test-time Adaptationcited by 57
  37. 2021Test-time Batch Statistics Calibration for Covariate Shiftcited by 50
  38. 2023AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuningcited by 49

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 30; 2025: 32; 2026: 25.

91 papers.

Sort:

By area:

  1. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  2. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 249
  3. 2024Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaborationcited by 176
  4. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 146
  5. 2024Analyzing the Generalization and Reliability of Steering Vectorscited by 130
  6. 2025Multi-Agent Design: Optimizing Agents with Better Prompts and Topologiescited by 106
  7. 2024Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvementscited by 82
  8. 2024Conformal Prediction for Natural Language Processing: A Surveycited by 52
  9. 2024Instruction Tuning With Loss Over Instructionscited by 40
  10. 2025The Prompt Makes the Person(a): A Systematic Evaluation of Sociodemographic Persona Prompting for Large Language Modelscited by 40
  11. 2024Thermometer: Towards Universal Calibration for Large Language Modelscited by 38
  12. 2025Investigating Non-Transitivity in LLM-as-a-Judgecited by 33
38 more
  1. 2024CalibraEval: Calibrating Prediction Distribution to Mitigate Selection Bias in LLMs-as-Judgescited by 32
  2. 2024Cycles of Thought: Measuring LLM Confidence through Stable Explanationscited by 27
  3. 2024Fairer Preferences Elicit Improved Human-Aligned Large Language Model Judgmentscited by 25
  4. 2024Beyond Performance: Quantifying and Mitigating Label Bias in LLMscited by 24
  5. 2023A Study on the Calibration of In-context Learningcited by 21
  6. 2024On Calibration of LLM-based Guard Models for Reliable Content Moderationcited by 21
  7. 2025Towards LLMs Robustness to Changes in Prompt Format Stylescited by 21
  8. 2025From Few to Many: Self-Improving Many-Shot Reasoners Through Iterative Optimization and Generationcited by 20
  9. 2025An Empirical Analysis of Uncertainty in Large Language Model Evaluationscited by 20
  10. 2026Large Language Model Reasoning Failurescited by 20
  11. 2025Maestro: Self-Improving Text-to-Image Generation via Agent Orchestrationcited by 18
  12. 2023Survival of the Most Influential Prompts: Efficient Black-Box Prompt Search via Clustering and Pruningcited by 17
  13. 2024Enhancing In-context Learning via Linear Probe Calibrationcited by 16
  14. 2025MAPRO: Recasting Multi-Agent Prompt Optimization as Maximum a Posteriori Inferencecited by 15
  15. 2023A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomnesscited by 14
  16. 2024Teaching LLMs to Abstain across Languages via Multilingual Feedbackcited by 14
  17. 2023Large Language Models are Miscalibrated In-Context Learnerscited by 12
  18. 2024Confidence Calibration of Classifiers with Many Classescited by 11
  19. 2025CausalAbstain: Enhancing Multilingual LLMs with Causal Reasoning for Trustworthy Abstentioncited by 10
  20. 2025When Punctuation Matters: A Large-Scale Comparison of Prompt Robustness Methods for LLMscited by 10
  21. 2024Unraveling the Mechanics of Learning-Based Demonstration Selection for In-Context Learningcited by 9
  22. 2024Token-based Decision Criteria Are Suboptimal in In-context Learningcited by 9
  23. 2024Rectifying Demonstration Shortcut in In-Context Learningcited by 7
  24. 2025Beyond the Next Token: Towards Prompt-Robust Zero-Shot Classification via Efficient Multi-Token Predictioncited by 7
  25. 2025What Really Matters in Many-Shot Attacks? An Empirical Study of Long-Context Vulnerabilities in LLMscited by 7
  26. 2025Conformal Information Pursuit for Interactively Guiding Large Language Modelscited by 7
  27. 2024When Parts Are Greater Than Sums: Individual LLM Components Can Outperform Full Modelscited by 6
  28. 2024Attention Instruction: Amplifying Attention in the Middle via Promptingcited by 6
  29. 2024Understanding Likelihood Over-optimisation in Direct Alignment Algorithmscited by 6
  30. 2025Calibrating LLM Judges: Linear Probes for Fast and Reliable Uncertainty Estimationcited by 6
  31. 2025RankAlign: A Ranking View of the Generator-Validator Gap in Large Language Modelscited by 4
  32. 2026From Words to Widgets for Controllable LLM Generationcited by 4
  33. 2025SafeRoute: Adaptive Model Selection for Efficient and Accurate Safety Guardrails in Large Language Modelscited by 3
  34. 2025Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Reviewcited by 3
  35. 2025Judging with Confidence: Calibrating Autoraters to Preference Distributionscited by 3
  36. 2026Revisiting Prompt Sensitivity in Large Language Models for Text Classification: The Role of Prompt Underspecificationcited by 3
  37. 2024Optimizing Class-Level Probability Reweighting Coefficients for Equitable Prompting Accuracycited by 2
  38. 2024Benchmarking Large Language Model Uncertainty for Prompt Optimizationcited by 2

50 most cited of 91 All 91 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Improving Calibration in Test-Time Prompt Tuning for Vision-Language Models via Data-Free Flatness-Aware Prompt Pretraining
  2. 2026Probabilistic Calibration Is a Trainable Capability in Language Models
  3. 2024Self-Instructed Derived Prompt Generation Meets In-Context Learning: Unlocking New Potential of Black-Box LLMs
  4. 2025Prompting without Panic: Attribute-aware, Zero-shot, Test-Time Calibration
  5. 2026What Single-Prompt Accuracy Misses: A Multi-Variant Reliability Audit of Language Models
  6. 2023Batch Prompting: Efficient Inference with Large Language Model APIs
  7. 2024C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersion
  8. 2026Towards Calibrating Prompt Tuning of Vision-Language Models