mlsanityData to 7 Oct 2026

Knowing More About Questions Can Help: Improving Calibration in Question Answering

arXiv 2106.01494 ↗First posted 2 Jun 2021cs.CL cs.AIcited by 47

Not on the map. Why?

Abstract (arXiv)

We study calibration in question answering, estimating whether model correctly predicts answer for each question. Unlike prior work which mainly rely on the model's confidence score, our calibrator incorporates information about the input example (e.g., question and the evidence context). Together with data augmentation via back translation, our simple approach achieves 5-10% gains in calibration accuracy on reading comprehension benchmarks. Furthermore, we present the first calibration study in the open retrieval setting, comparing the calibration accuracy of retrieval-based span prediction models and answer generation models. Here again, our approach shows consistent gains over calibrators relying on the model confidence. Our simple and efficient calibrator can be easily adapted to many tasks and model architectures, showing robust gains in all settings.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 44 of its 58 references that are in the map. First posted on arXiv 2 Jun 2021.

Builds on

44 of 58 references.

References · 44

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  6. 2017On Calibration of Modern Neural Networkscited by 4,860
  7. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  9. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  10. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  11. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  12. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
32 more
  1. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  2. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  3. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  4. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  5. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  6. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  7. 2017On Fairness and Calibrationcited by 406
  8. 2019Using Pre-Training Can Improve Model Robustness and Uncertaintycited by 373
  9. 2016NewsQA: A Machine Comprehension Datasetcited by 330
  10. 2020Calibration of Pre-trained Transformerscited by 307
  11. 2016Bidirectional Attention Flow for Machine Comprehensioncited by 278
  12. 2017Detecting Adversarial Samples from Artifactscited by 244
  13. 2020Selective Question Answering under Domain Shiftcited by 228
  14. 2017SearchQA: A New Q&A Dataset Augmented with Context from a Search Enginecited by 221
  15. 2018Marian: Fast Neural Machine Translation in C++cited by 202
  16. 2019MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehensioncited by 183
  17. 2018Understanding Measures of Uncertainty for Adversarial Example Detectioncited by 177
  18. 2019Not Enough Data? Deep Learning to the Rescue!cited by 174
  19. 2018QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehensioncited by 164
  20. 2020Self-training Improves Pre-training for Natural Language Understandingcited by 96
  21. 2015Posterior calibration and exploratory analysis for natural language processing modelscited by 75
  22. 2019Calibration of Encoder Decoder Models for Neural Machine Translationcited by 67
  23. 2017Learning to Paraphrase for Question Answeringcited by 46
  24. 2019XLDA: Cross-Lingual Data Augmentation for Natural Language Inference and Question Answeringcited by 35
  25. 2018Confidence Modeling for Neural Semantic Parsingcited by 33
  26. 2019A BERT Baseline for the Natural Questionscited by 33
  27. 2018Read + Verify: Machine Reading Comprehension with Unanswerable Questionscited by 31
  28. 2021Contextual Dropout: An Efficient Sample-Dependent Dropout Modulecited by 26
  29. 2020Diverse Ensembles Improve Calibrationcited by 23
  30. 2020Calibrating Structured Output Predictors for Natural Language Processingcited by 17
  31. 2020No Answer is Better Than Wrong Answer: A Reflection Model for Document Level Machine Reading Comprehensioncited by 7
  32. 2019Unsupervised Out-of-Distribution Detection with Batch Normalizationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 11; 2023: 17; 2024: 10; 2025: 3; 2026: 1.

47 papers.

Sort:

By area:

  1. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 253
  2. 2022The Unreliability of Explanations in Few-shot Prompting for Textual Reasoningcited by 169
  3. 2022Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidencecited by 142
  4. 2023Document Understanding Dataset and Evaluation (DUDE)cited by 132
  5. 2023AutoML-GPT: Automatic Machine Learning with GPTcited by 80
  6. 2024Learning to Route LLMs with Confidence Tokenscited by 66
  7. 2023Selectively Answering Ambiguous Questionscited by 63
  8. 2022A Close Look into the Calibration of Pre-trained Language Modelscited by 57
  9. 2023Why So Gullible? Enhancing the Robustness of Retrieval-Augmented Models against Counterfactual Noisecited by 52
  10. 2023Uncertainty in Natural Language Processing: Sources, Quantification, and Applicationscited by 44
  11. 2024Thermometer: Towards Universal Calibration for Large Language Modelscited by 38
  12. 2023Learning to Reject with a Fixed Predictor: Application to Decontextualizationcited by 32
35 more
  1. 2023Preference-grounded Token-level Guidance for Language Model Fine-tuningcited by 31
  2. 2021Learning with Different Amounts of Annotation: From Zero to Many Labelscited by 28
  3. 2025Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approachcited by 25
  4. 2022Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcitycited by 24
  5. 2022ALLSH: Active Learning Guided by Local Sensitivity and Hardnesscited by 23
  6. 2023Mitigating Temporal Misalignment by Discarding Outdated Factscited by 21
  7. 2021Can Explanations Be Useful for Calibrating Black Box Models?cited by 20
  8. 2023Post-Abstention: Towards Reliably Re-Attempting the Abstained Instances in QAcited by 20
  9. 2021Will this Question be Answered? Question Filtering via Answer Model Distillation for Efficient Question Answeringcited by 19
  10. 2022On the Limitations of Dataset Balancing: The Lost Battle Against Spurious Correlationscited by 19
  11. 2023Fantastic Rewards and How to Tame Them: A Case Study on Reward Learning for Task-oriented Dialogue Systemscited by 19
  12. 2023CATfOOD: Counterfactual Augmented Training for Improving Out-of-Domain Performance and Calibrationcited by 16
  13. 2024The "Colonial Impulse" of Natural Language Processing: An Audit of Bengali Sentiment Analysis Tools and Their Identity-based Biasescited by 16
  14. 2024Regurgitative Training: The Value of Real Data in Training Large Language Modelscited by 15
  15. 2021Alignment Attention by Matching Key and Query Distributionscited by 14
  16. 2024Large Language Model Confidence Estimation via Black-Box Accesscited by 14
  17. 2024Crowd-Calibrator: Can Annotator Disagreement Inform Calibration in Subjective Tasks?cited by 13
  18. 2023Simple Token-Level Confidence Improves Caption Correctnesscited by 11
  19. 2025Uncertainty Quantification in Retrieval Augmented Question Answeringcited by 8
  20. 2022Passage-Mask: A Learnable Regularization Strategy for Retriever-Reader Modelscited by 6
  21. 2023On the Calibration of Multilingual Question Answering LLMscited by 6
  22. 2024Deal, or no deal (or who knows)? Forecasting Uncertainty in Conversations using Large Language Modelscited by 6
  23. 2023From Simple to Complex: A Progressive Framework for Document-level Informative Argument Extractioncited by 4
  24. 2024Language Rectified Flow: Advancing Diffusion Language Generation with Probabilistic Flowscited by 4
  25. 2023When to Read Documents or QA History: On Unified and Selective Open-domain QAcited by 3
  26. 2022Understanding Prior Bias and Choice Paralysis in Transformer-based Language Representation Models through Four Experimental Probescited by 2
  27. 2023A Prototype-Oriented Clustering for Domain Shift with Source Privacycited by 2
  28. 2025Shapley Uncertainty in Natural Language Generationcited by 2
  29. 2022Contrastive Novelty-Augmented Learning: Anticipating Outliers with Large Language Modelscited by 1
  30. 2021Interviewer-Candidate Role Play: Towards Developing Real-World NLP Systemscited by 0
  31. 2022Using calibrator to improve robustness in Machine Reading Comprehensioncited by 0
  32. 2022Re-Examining Calibration: The Case of Question Answeringcited by 0
  33. 2024ARTiST: Automated Text Simplification for Task Guidance in Augmented Realitycited by 0
  34. 2024Switchable Decision: Dynamic Neural Generation Networkscited by 0
  35. 2026Beyond Post-Hoc Temperature Scaling: Bilevel Optimization for LLM Calibrationcited by 0