mlsanityData to 7 Oct 2026

Investigating Selective Prediction Approaches Across Several Tasks in IID, OOD, and Adversarial Settings

arXiv 2203.00211 ↗First posted 1 Mar 2022cs.CL cs.AI cs.LGcited by 66

Not on the map. Why?

Abstract (arXiv)

In order to equip NLP systems with selective prediction capability, several task-specific approaches have been proposed. However, which approaches work best across tasks or even if they consistently outperform the simplest baseline 'MaxProb' remains to be explored. To this end, we systematically study 'selective prediction' in a large-scale setup of 17 datasets across several NLP tasks. Through comprehensive experiments under in-domain (IID), out-of-domain (OOD), and adversarial (ADV) settings, we show that despite leveraging additional resources (held-out data/computation), none of the existing approaches consistently and considerably outperforms MaxProb in all three settings. Furthermore, their performance does not translate well across tasks. For instance, Monte-Carlo Dropout outperforms all other approaches on Duplicate Detection datasets but does not fare well on NLI datasets, especially in the OOD setting. Thus, we recommend that future selective prediction approaches should be evaluated across tasks and settings for reliable estimation of their capabilities.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). Builds on 27 of its 37 references that are in the map. First posted on arXiv 1 Mar 2022.

Builds on

27 of 37 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  9. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  10. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  11. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  12. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
15 more
  1. 2017Selective Classification for Deep Neural Networkscited by 799
  2. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  3. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
  4. 2019SelectiveNet: A Deep Neural Network with an Integrated Reject Optioncited by 333
  5. 2016NewsQA: A Machine Comprehension Datasetcited by 330
  6. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 293
  7. 2020Selective Question Answering under Domain Shiftcited by 228
  8. 2017SearchQA: A New Q&A Dataset Augmented with Context from a Search Enginecited by 221
  9. 2018Stress Test Evaluation for Natural Language Inferencecited by 211
  10. 2018Dialogue Natural Language Inferencecited by 135
  11. 2018Breaking NLI Systems with Sentences that Require Simple Lexical Inferencescited by 134
  12. 2020The Effect of Natural Distribution Shift on Question Answering Modelscited by 104
  13. 2019MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehensioncited by 60
  14. 2021Will this Question be Answered? Question Filtering via Answer Model Distillation for Efficient Question Answeringcited by 19
  15. 2020Our Evaluation Metric Needs an Update to Encourage Generalizationcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 14; 2023: 23; 2024: 13; 2025: 4; 2026: 12.

66 papers.

Sort:

By area:

  1. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  2. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  3. 2023Generating with Confidence: Uncertainty Quantification for Black-box Large Language Modelscited by 286
  4. 2023A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generationcited by 202
  5. 2023LM vs LM: Detecting Factual Errors via Cross Examinationcited by 155
  6. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  7. 2022Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysiscited by 104
  8. 2024The Art of Saying No: Contextual Noncompliance in Language Modelscited by 100
  9. 2022DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answeringcited by 93
  10. 2022Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectlycited by 86
  11. 2023Crawling the Internal Knowledge-Base of Language Modelscited by 85
  12. 2022GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspectivecited by 72
38 more
  1. 2023Selectively Answering Ambiguous Questionscited by 63
  2. 2023The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensivenesscited by 63
  3. 2022A Close Look into the Calibration of Pre-trained Language Modelscited by 57
  4. 2024Conformal Alignment: Knowing When to Trust Foundation Models with Guaranteescited by 50
  5. 2023Do Large Language Models Know about Facts?cited by 49
  6. 2023Adaptation with Self-Evaluation to Improve Selective Prediction in LLMscited by 49
  7. 2024Distinguishing the Knowable from the Unknowable with Language Modelscited by 45
  8. 2023Uncertainty in Natural Language Processing: Sources, Quantification, and Applicationscited by 44
  9. 2022Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systemscited by 43
  10. 2022Calibrated Interpretation: Confidence Estimation in Semantic Parsingcited by 33
  11. 2023A Survey on Out-of-Distribution Detection in NLPcited by 31
  12. 2024I Don't Know: Explicit Modeling of Uncertainty with an [IDK] Tokencited by 26
  13. 2023Improving Selective Visual Question Answering by Learning from Your Peerscited by 24
  14. 2022Reasoning over Public and Private Data in Retrieval-Based Systemscited by 21
  15. 2023Post-Abstention: Towards Reliably Re-Attempting the Abstained Instances in QAcited by 20
  16. 2023Measuring and Modifying Factual Knowledge in Large Language Modelscited by 16
  17. 2024Contextualized Sequence Likelihood: Enhanced Confidence Scores for Natural Language Generationcited by 16
  18. 2024MFC-Bench: Benchmarking Multimodal Fact-Checking with Large Vision-Language Modelscited by 15
  19. 2024Certainly Uncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awarenesscited by 13
  20. 2023Simple Token-Level Confidence Improves Caption Correctnesscited by 11
  21. 2026Calibrated Decisions at Scale: Converting Police Crash Narratives into Probabilistic Crash Variables with a System One Model (Jev)cited by 11
  22. 2022Can Open-Domain QA Reader Utilize External Knowledge Efficiently like Humans?cited by 10
  23. 2023Outlier Robust Adversarial Trainingcited by 10
  24. 2023Learning to Generalize for Cross-domain QAcited by 6
  25. 2024Out-of-Distribution Data: An Acquaintance of Adversarial Examples -- A Surveycited by 6
  26. 2023Did You Mean...? Confidence-based Trade-offs in Semantic Parsingcited by 5
  27. 2023Zero-Shot End-to-End Spoken Language Understanding via Cross-Modal Selective Self-Trainingcited by 5
  28. 2024Efficiently Deploying LLMs with Controlled Riskcited by 5
  29. 2022Discover, Explanation, Improvement: An Automatic Slice Detection Framework for Natural Language Processingcited by 4
  30. 2023Can NLP Models 'Identify', 'Distinguish', and 'Justify' Questions that Don't have a Definitive Answer?cited by 4
  31. 2026Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Modelscited by 4
  32. 2023Finding Competence Regions in Domain Generalizationcited by 3
  33. 2025InFact: Informativeness Alignment for Improved LLM Factualitycited by 3
  34. 2025Detecting AI Hallucinations in Finance: An Information-Theoretic Method Cuts Hallucination Rate by 92%cited by 2
  35. 2026Knowing When to Answer: Adaptive Confidence Refinement for Reliable Audio-Visual Question Answeringcited by 2
  36. 2022Hardness of Samples Need to be Quantified for a Reliable Evaluation System: Exploring Potential Opportunities with a New Taskcited by 1
  37. 2022Contrastive Novelty-Augmented Learning: Anticipating Outliers with Large Language Modelscited by 1
  38. 2023A Unified Evaluation Framework for Novelty Detection and Accommodation in NLP with an Instantiation in Authorship Attributioncited by 1

50 most cited of 66 All 66 (JSON)