mlsanityData to 7 Oct 2026

Selectively Answering Ambiguous Questions

arXiv 2305.14613 ↗First posted 24 May 2023cs.CL cs.AIcited by 63

Not on the map. Why?

Abstract (arXiv)

Trustworthy language models should abstain from answering questions when they do not know the answer. However, the answer to a question can be unknown for a variety of reasons. Prior research has focused on the case in which the question is clear and the answer is unambiguous but possibly unknown, but the answer to a question can also be unclear due to uncertainty of the questioner's intent or context. We investigate question answering from this perspective, focusing on answering a subset of questions with a high degree of accuracy, from a set of questions in which many are inherently ambiguous. In this setting, we find that the most reliable approach to decide when to abstain involves quantifying repetition within sampled model outputs, rather than the model's likelihood or self-verification as used in prior work. We find this to be the case across different types of uncertainty and model scales,and with or without instruction tuning. Our results suggest that sampling-based confidence scores help calibrate answers to relatively unambiguous questions, with more dramatic improvements on ambiguous questions.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Builds on 30 of its 44 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

30 of 44 references.

References · 30

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2017On Calibration of Modern Neural Networkscited by 4,860
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  7. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  8. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  9. 2018Hierarchical Neural Story Generationcited by 1,182
  10. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  11. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  12. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
18 more
  1. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  2. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  3. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 385
  4. 2020Calibration of Pre-trained Transformerscited by 307
  5. 2022ASQA: Factoid Questions Meet Long-Form Answerscited by 231
  6. 2020Selective Question Answering under Domain Shiftcited by 228
  7. 2022Out-of-Distribution Detection and Selective Generation for Conditional Language Modelscited by 193
  8. 2021SituatedQA: Incorporating Extra-Linguistic Contexts into QAcited by 145
  9. 2021Epistemic Neural Networkscited by 120
  10. 2022Locally Typical Samplingcited by 108
  11. 2019Calibration of Encoder Decoder Models for Neural Machine Translationcited by 67
  12. 2022Investigating Selective Prediction Approaches Across Several Tasks in IID, OOD, and Adversarial Settingscited by 66
  13. 2021Knowing More About Questions Can Help: Improving Calibration in Question Answeringcited by 47
  14. 2022Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systemscited by 43
  15. 2018Confidence Modeling for Neural Semantic Parsingcited by 33
  16. 2020Calibrating Structured Output Predictors for Natural Language Processingcited by 17
  17. 2021Conditional Poisson Stochastic Beam Searchcited by 11
  18. 2022Estimating the Entropy of Linguistic Distributionscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 19; 2025: 19; 2026: 21.

63 papers.

Sort:

By area:

  1. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 246
  2. 2024Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similaritiescited by 156
  3. 2023Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselvescited by 123
  4. 2024To Believe or Not to Believe Your LLMcited by 120
  5. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  6. 2023Decomposing Uncertainty for Large Language Models through Input Clarification Ensemblingcited by 113
  7. 2023Self-Evaluation Improves Selective Generation in Large Language Modelscited by 74
  8. 2024Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questionscited by 70
  9. 2023Uncertainty in Natural Language Generation: From Theory to Applicationscited by 61
  10. 2025Missing Premise exacerbates Overthinking: Are Reasoning Models losing Critical Thinking Skill?cited by 58
  11. 2024Mitigating LLM Hallucinations via Conformal Abstentioncited by 56
  12. 2024Calibrating Long-form Generations from Large Language Modelscited by 31
38 more
  1. 2025Estimating LLM Uncertainty with Evidencecited by 26
  2. 2024MAQA: Evaluating Uncertainty Quantification in LLMs Regarding Data Uncertaintycited by 25
  3. 2024Finetuning Language Models to Emit Linguistic Expressions of Uncertaintycited by 25
  4. 2024AmbigDocs: Reasoning across Documents on Different Entities under the Same Namecited by 21
  5. 2025From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centeredcited by 21
  6. 2024Estimating the Hallucination Rate of Generative AIcited by 20
  7. 2025Semantic Volume: Quantifying and Detecting both External and Internal Uncertainty in LLMscited by 19
  8. 2025Uncertainty Quantification for Language Models: A Suite of Black-Box, White-Box, LLM Judge, and Ensemble Scorerscited by 16
  9. 2026Calibrate-Then-Act: Cost-Aware Exploration in LLM Agentscited by 16
  10. 2024Crowd-Calibrator: Can Annotator Disagreement Inform Calibration in Subjective Tasks?cited by 13
  11. 2025Disambiguate First, Parse Later: Generating Interpretations for Ambiguity Resolution in Semantic Parsingcited by 12
  12. 2024Varying Shades of Wrong: Aligning LLMs with Wrong Answers Onlycited by 9
  13. 2025Grammars of Formal Uncertainty: When to Trust LLMs in Automated Reasoning Taskscited by 9
  14. 2025UQLM: A Python Package for Uncertainty Quantification in Large Language Modelscited by 9
  15. 2024Can Tool-augmented Large Language Models be Aware of Incomplete Conditions?cited by 8
  16. 2024Is There No Such Thing as a Bad Question? H4R: HalluciBot For Ratiocination, Rewriting, Ranking, and Routingcited by 6
  17. 2024Improving Instruction Following in Language Models through Proxy-Based Uncertainty Estimationcited by 6
  18. 2025The Consistency Hypothesis in Uncertainty Quantification for Large Language Modelscited by 6
  19. 2025Learning Steerable Clarification Policies with Collaborative Self-playcited by 5
  20. 2026The Cost of Reasoning: Chain-of-Thought Induces Overconfidence in Vision-Language Modelscited by 5
  21. 2026Fine-Grained Uncertainty Quantification for Long-Form Language Model Outputs: A Comparative Studycited by 4
  22. 2024CSS: Contrastive Semantic Similarity for Uncertainty Quantification of LLMscited by 3
  23. 2024LLMs Can Check Their Own Results to Mitigate Hallucinations in Traffic Understanding Taskscited by 3
  24. 2025"It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Modelscited by 3
  25. 2026Architecting Trust in Artificial Epistemic Agentscited by 3
  26. 2026MEMORA: Embodied Action Memory from Egocentric Videos for Reasoning and Planningcited by 3
  27. 2024Can Generative AI Solve Your In-Context Learning Problem? A Martingale Perspectivecited by 2
  28. 2025Know Or Not: a library for evaluating out-of-knowledge base robustnesscited by 2
  29. 2025Generation Space Size: Understanding and Calibrating Open-Endedness of LLM Generationscited by 2
  30. 2025SIMBA UQ: Similarity-Based Aggregation for Uncertainty Quantification in Large Language Modelscited by 2
  31. 2026Diagnosing Multi-step Reasoning Failures in Black-box LLMs via Stepwise Confidence Attributioncited by 2
  32. 2026Credal Large Language Models for Semantic Commitment under Uncertaintycited by 2
  33. 2025Dist2ill: Distributional Distillation for One-Pass Uncertainty Estimation in Large Language Modelscited by 1
  34. 2025QueryBandits for Hallucination Mitigation: Exploiting Semantic Features for No-Regret Rewritingcited by 1
  35. 2025Rule-Based Moral Principles for Explaining Uncertainty in Natural Language Generationcited by 1
  36. 2025Learning to Extract Context for Context-Aware LLM Inferencecited by 1
  37. 2026Anatomy of a Lie: A Multi-Stage Diagnostic Framework for Tracing Hallucinations in Vision-Language Modelscited by 1
  38. 2026Knowing but Not Showing: LLMs Recognize Ambiguity but Rarely Ask Clarifying Questionscited by 1

50 most cited of 63 All 63 (JSON)