mlsanityData to 14 Oct 2026

LitCab: Lightweight Language Model Calibration over Short- and Long-form Responses

arXiv 2310.19208 ↗First posted 30 Oct 2023cs.CLcited by 44

Not on the map. Why?

Abstract (arXiv)

A model is considered well-calibrated when its probability estimate aligns with the actual likelihood of the output being correct. Calibrating language models (LMs) is crucial, as it plays a vital role in detecting and mitigating hallucinations of LMs as well as building more trustworthy models. However, standard calibration techniques may not be suited for LM calibration. For instance, post-processing methods such as temperature scaling do not reorder the candidate generations. On the other hand, training-based methods require fine-tuning the entire model, which is impractical for LMs of large scale. We present LitCab, a lightweight calibration mechanism consisting of a single linear layer that takes the input text representation and predicts a bias term, which is then added to the LM output logits. LitCab improves model calibration by only adding < 2% of the original model parameters. For evaluation, we construct CaT, a benchmark consisting of eight text generation tasks, covering responses ranging from short phrases to paragraphs. We test LitCab with Llama2-7B, where it improves calibration across all tasks, reducing the average ECE score by as large as 30%. We further conduct a comprehensive evaluation with multiple popular open-sourced LMs from GPT and LLaMA families, yielding the following key findings: (i) Larger models within the same family exhibit better calibration on tasks with short generation tasks, but not necessarily for longer ones. (ii) GPT-family models show superior calibration compared to LLaMA, Llama2, and Vicuna models, despite having much fewer parameters. (iii) Fine-tuning pretrained model (e.g., LLaMA) with samples of limited purpose (e.g., conversations) may lead to worse calibration, highlighting the importance of fine-tuning setups for calibrating LMs.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Builds on 28 of its 33 references that are in the map. First posted on arXiv 30 Oct 2023.

Builds on

28 of 33 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  7. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  8. 2017On Calibration of Modern Neural Networkscited by 4,960
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  10. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  11. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  12. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
16 more
  1. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,328
  2. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  3. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  4. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  5. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  6. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  7. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 464
  8. 2021Large Dual Encoders Are Generalizable Retrieverscited by 456
  9. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 445
  10. 2022Prompting GPT-3 To Be Reliablecited by 260
  11. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 236
  12. 2020Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learningcited by 190
  13. 2015Posterior calibration and exploratory analysis for natural language processing modelscited by 76
  14. 2019Non-Parametric Calibration for Classificationcited by 57
  15. 2022QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphscited by 42
  16. 2016Neural Text Generation from Structured Data with Application to the Biography Domaincited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 11; 2025: 19; 2026: 14.

44 papers.

Sort:

By area:

  1. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 121
  2. 2025Process Reward Models That Thinkcited by 118
  3. 2025Uncertainty Quantification and Confidence Calibration in Large Language Models: A Surveycited by 95
  4. 2024Calibration and Correctness of Language Models for Codecited by 49
  5. 2025Mind the Confidence Gap: Overconfidence, Calibration, and Distractor Effects in Large Language Modelscited by 40
  6. 2024Confidence Calibration and Rationalization for LLMs via Multi-Agent Deliberationcited by 26
  7. 2024Open-RAG: Enhanced Retrieval-Augmented Reasoning with Open-Source Large Language Modelscited by 25
  8. 2025SimPER: A Minimalist Approach to Preference Alignment without Hyperparameterscited by 25
  9. 2024Atomic Calibration of LLMs in Long-Form Generationscited by 24
  10. 2025Your Pre-trained LLM is Secretly an Unsupervised Confidence Calibratorcited by 20
  11. 2025Mitigating LLM Hallucination via Behaviorally Calibrated Reinforcement Learningcited by 18
  12. 2025LoVeC: Reinforcement Learning for Better Verbalized Confidence in Long-Form Generationscited by 17
32 more
  1. 2025MetaFaith: Faithful Natural Language Uncertainty Expression in LLMscited by 16
  2. 2024Enhancing Language Model Factuality via Activation-Based Confidence Calibration and Guided Decodingcited by 14
  3. 2024Generative AI for Education (GAIED): Advances, Opportunities, and Challengescited by 13
  4. 2025CausalAbstain: Enhancing Multilingual LLMs with Causal Reasoning for Trustworthy Abstentioncited by 10
  5. 2024A Survey of Calibration Process for Black-Box LLMscited by 8
  6. 2024Enhanced Language Model Truthfulness with Learnable Intervention and Uncertainty Expressioncited by 6
  7. 2025COPU: Conformal Prediction for Uncertainty Quantification in Natural Language Generationcited by 6
  8. 2025B-score: Detecting biases in large language models using response historycited by 6
  9. 2024Enhancing Trust in Large Language Models via Uncertainty-Calibrated Fine-Tuningcited by 5
  10. 2025Controlling Large Language Model with Latent Actionscited by 5
  11. 2025Uncertainty Unveiled: Can Exposure to More In-context Examples Mitigate Uncertainty for Large Language Models?cited by 5
  12. 2025Generalized Correctness Models: Learning Calibrated and Model-Agnostic Correctness Predictors from Historical Patternscited by 5
  13. 2025Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Reviewcited by 4
  14. 2026Influential Training Data Retrieval for Explaining Verbalized Confidence of LLMscited by 4
  15. 2026Fine-Grained Uncertainty Quantification for Long-Form Language Model Outputs: A Comparative Studycited by 4
  16. 2024Semantic Density: Uncertainty Quantification for Large Language Models through Confidence Measurement in Semantic Spacecited by 3
  17. 2026Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variationscited by 3
  18. 2025Eliminating Hallucination-Induced Errors in LLM Code Generation with Functional Clusteringcited by 2
  19. 2025REHEARSE: Experiential Rehearsal for Verbal Confidence Calibration in Large Language Modelscited by 2
  20. 2026Think Through Uncertainty: Improving Long-Form Generation Factuality via Reasoning Calibrationcited by 2
  21. 2025HalluCounter: Reference-free LLM Hallucination Detection in the Wild!cited by 1
  22. 2026Unlocking the Pre-Trained Model as a Dual-Alignment Calibrator for Post-Trained LLMscited by 1
  23. 2026Self-Calibrating Language Models via Test-Time Discriminative Distillationcited by 1
  24. 2026Shared Doubt: Zero-Shot Cross-Lingual Confidence Estimation for Language Modelscited by 1
  25. 2025FALCONEye: Finding Answers and Localizing Content in ONE-hour-long videos with multi-modal LLMscited by 0
  26. 2026Fine-grained Approaches for Confidence Calibration of LLMs in Automated Code Revisioncited by 0
  27. 2026Assessing and Mitigating Miscalibration in LLM-Based Social Science Measurementcited by 0
  28. 2026ECUASn: A family of metrics for principled evaluation of uncertainty-augmented systemscited by 0
  29. 2026Vision-Language Model Confidence Is Not a Property of the Answercited by 0
  30. 2026Beyond Post-Hoc Temperature Scaling: Bilevel Optimization for LLM Calibrationcited by 0
  31. 2026Claim-Level Confidence Calibration for Reliable Decision Making with Large Language Modelscited by 0
  32. 2026On the Pitfalls of Verbalized Confidence Priors for Calibrating Large Reasoning Modelscited by 0