mlsanityData to 14 Oct 2026

Selective Question Answering under Domain Shift

arXiv 2006.09462 ↗First posted 16 Jun 2020cs.CL cs.LGcited by 238

Not on the map. Why?

Abstract (arXiv)

To avoid giving wrong answers, question answering (QA) models need to know when to abstain from answering. Moreover, users often ask questions that diverge from the model's training data, making errors more likely and thus abstention more critical. In this work, we propose the setting of selective question answering under domain shift, in which a QA model is tested on a mixture of in-domain and out-of-domain data, and must answer (i.e., not abstain on) as many questions as possible while maintaining high accuracy. Abstention policies based solely on the model's softmax probabilities fare poorly, since models are overconfident on out-of-domain inputs. Instead, we train a calibrator to identify inputs on which the QA model errs, and abstain when it predicts an error is likely. Crucially, the calibrator benefits from observing the model's behavior on out-of-domain data, even if from a different domain than the test data. We combine this method with a SQuAD-trained QA model and evaluate on mixtures of SQuAD and five other QA datasets. Our method answers 56% of questions while maintaining 80% accuracy; in contrast, directly using the model's probabilities only answers 48% at 80% accuracy.

Key facts. Cited by 238 papers in the map (data through 14 Oct 2026). First posted on arXiv 16 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 21; 2022: 34; 2023: 48; 2024: 39; 2025: 28; 2026: 68.

238 papers.

Sort:

By area:

  1. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  2. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  3. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 401
  4. 2023Generating with Confidence: Uncertainty Quantification for Black-box Large Language Modelscited by 293
  5. 2022Teaching language models to support answers with verified quotescited by 292
  6. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 256
  7. 2023A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generationcited by 206
  8. 2024Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaborationcited by 180
  9. 2022The Unreliability of Explanations in Few-shot Prompting for Textual Reasoningcited by 169
  10. 2023LM vs LM: Detecting Factual Errors via Cross Examinationcited by 156
  11. 2022Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidencecited by 144
  12. 2023Document Understanding Dataset and Evaluation (DUDE)cited by 135
38 more
  1. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 121
  2. 2024Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertaintycited by 106
  3. 2024The Art of Saying No: Contextual Noncompliance in Language Modelscited by 105
  4. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 98
  5. 2023Clarify When Necessary: Resolving Ambiguity Through Interaction with LMscited by 98
  6. 2023Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Modelscited by 91
  7. 2022Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectlycited by 89
  8. 2024Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words?cited by 87
  9. 2021Extending the WILDS Benchmark for Unsupervised Adaptationcited by 86
  10. 2024LUQ: Long-text Uncertainty Quantification for LLMscited by 81
  11. 2021Types of Out-of-Distribution Texts and How to Detect Themcited by 80
  12. 2022On the Relation between Sensitivity and Accuracy in In-context Learningcited by 72
  13. 2021Break-It-Fix-It: Unsupervised Learning for Program Repaircited by 71
  14. 2021Revisiting Mahalanobis Distance for Transformer-Based Out-of-Domain Detectioncited by 70
  15. 2023The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensivenesscited by 67
  16. 2024Learning to Route LLMs with Confidence Tokenscited by 67
  17. 2022Investigating Selective Prediction Approaches Across Several Tasks in IID, OOD, and Adversarial Settingscited by 66
  18. 2023Selectively Answering Ambiguous Questionscited by 66
  19. 2021Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answeringcited by 62
  20. 2023Uncertainty in Natural Language Generation: From Theory to Applicationscited by 61
  21. 2024Mitigating LLM Hallucinations via Conformal Abstentioncited by 59
  22. 2023Why So Gullible? Enhancing the Robustness of Retrieval-Augmented Models against Counterfactual Noisecited by 54
  23. 2024Conformal Alignment: Knowing When to Trust Foundation Models with Guaranteescited by 54
  24. 2022A Call to Reflect on Evaluation Practices for Failure Detection in Image Classificationcited by 52
  25. 2021Knowing More About Questions Can Help: Improving Calibration in Question Answeringcited by 47
  26. 2024Distinguishing the Knowable from the Unknowable with Language Modelscited by 47
  27. 2021Can NLI Models Verify QA Systems' Predictions?cited by 46
  28. 2024Characterizing Truthfulness in Large Language Model Generations with Local Intrinsic Dimensioncited by 45
  29. 2023Uncertainty in Natural Language Processing: Sources, Quantification, and Applicationscited by 44
  30. 2021Attacking Open-domain Question Answering by Injecting Misinformationcited by 41
  31. 2023Llamas Know What GPTs Don't Show: Surrogate Models for Confidence Estimationcited by 39
  32. 2025No Answer Needed: Predicting LLM Answer Accuracy from Question-Only Linear Probescited by 38
  33. 2022Uncertainty Estimation for Language Reward Modelscited by 37
  34. 2021FaVIQ: FAct Verification from Information-seeking Questionscited by 36
  35. 2022Calibrated Selective Classificationcited by 33
  36. 2023Learning to Reject with a Fixed Predictor: Application to Decontextualizationcited by 32
  37. 2024Calibrating Long-form Generations from Large Language Modelscited by 32
  38. 2024Consistency and Uncertainty: Identifying Unreliable Responses From Black-Box Vision-Language Models for Selective Visual Question Answeringcited by 32

50 most cited of 238 All 238 (JSON)