Towards Universal Calibration for In-Context Classification
Not on the map. Why?
Abstract (arXiv)
In-Context Learning (ICL) has received a lot of interest recently due to its ability to improve classification accuracy for new tasks without having to fine-tune the model over these tasks. However, the impact of ICL on calibration has been far less studied. Recent work has shown that ICL can have a negative impact on calibration due to biased selection of in-context examples. The existing methods for improving ICL calibration require additional task and model-specific training, detracting from the no-fine-tuning convenience of ICL. In this work, we introduce CALIBFORMER, a universal ICL calibrator for classification tasks, that generalizes to new models and tasks without further fine-tuning on these tasks. Specifically, we design a transformer-based calibrator which uses the model predictions on in-context examples to predict a confidence score for the test input. We show that the calibrator trained on three models and ten tasks, is able to generalize/calibrate ICL predictions on several new tasks and four LLMs unseen during training. Even for frontier models like Gemini, CALIBFORMER is able to obtain over 50% reduction in calibration error.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 26 of its 37 references that are in the map. First posted on arXiv 12 Oct 2026.
Builds on
26 of 37 references.
References · 26
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
- 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
- 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
- 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
- 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
- 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 496
- 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 217
- 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 161
- 2024Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreementcited by 105
- 2023The Transient Nature of Emergent In-Context Learning in Transformerscited by 76
14 more
- 2024Calibrating Large Language Models Using Their Generations Onlycited by 65
- 2023Smooth ECE: Principled Reliability Diagrams via Kernel Smoothingcited by 63
- 2023Steering Large Language Models for Machine Translation with Finetuning and In-Context Learningcited by 49
- 2021I Wish I Would Have Loved This One, But I Didn't -- A Multilingual Dataset for Counterfactual Detection in Product Reviewscited by 45
- 2025When2Call: When (not) to Call Toolscited by 42
- 2024Evaluating language models as risk scorescited by 27
- 2023A Study on the Calibration of In-context Learningcited by 22
- 2020AutoClip: Adaptive Gradient Clipping for Source Separation Networkscited by 12
- 2024Rectifying Demonstration Shortcut in In-Context Learningcited by 9
- 2025It's All About In-Context Learning! Teaching Extremely Low-Resource Languages to LLMscited by 6
- 2025Understanding the Generalization of In-Context Learning in Transformers: An Empirical Studycited by 4
- 2025Improving Large Language Models Function Calling and Interpretability via Guided-Structured Templatescited by 4
- 2025In-Context Learning Enhanced Credibility Transformercited by 3
- 2025Surprise Calibration for Better In-Context Learningcited by 2
Built on by
Not cited yet.