# Question Difficulty Estimation for Large Language Models via Answer Plausibility Scoring

arXiv: https://arxiv.org/abs/2605.12398 · HTML: https://mlsanity.com/paper/2605.12398/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 59 references that are in the map.
- First posted on arXiv 12 May 2026.

## Builds on (most cited first, 26 of 26)

- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [AmbigQA: Answering Ambiguous Open-domain Questions](https://mlsanity.com/paper/2004.10645/) (2020)
- [QASC: A Dataset for Question Answering via Sentence Composition](https://mlsanity.com/paper/1910.11473/) (2019)
- [Simple Entity-Centric Questions Challenge Dense Retrievers](https://mlsanity.com/paper/2109.08535/) (2021)
- [Unlocking Context Constraints of LLMs: Enhancing Context Efficiency of LLMs with Self-Information-Based Content Filtering](https://mlsanity.com/paper/2304.12102/) (2023)
- [A Survey on Popularity Bias in Recommender Systems](https://mlsanity.com/paper/2308.01118/) (2023)
- [When to Trust LLMs: Aligning Confidence with Response Quality](https://mlsanity.com/paper/2404.17287/) (2024)
- [MQAG: Multiple-choice Question Answering and Generation for Assessing Information Consistency in Summarization](https://mlsanity.com/paper/2301.12307/) (2023)
- [ComQA: A Community-sourced Dataset for Complex Factoid Question Answering with Paraphrase Clusters](https://mlsanity.com/paper/1809.09528/) (2018)
- [Distractor Generation in Multiple-Choice Tasks: A Survey of Methods, Datasets, and Evaluation](https://mlsanity.com/paper/2402.01512/) (2024)
- [CLARINET: Augmenting Language Models to Ask Clarification Questions for Retrieval](https://mlsanity.com/paper/2405.15784/) (2024)
- [ReadMe++: Benchmarking Multilingual Language Models for Multi-Domain Readability Assessment](https://mlsanity.com/paper/2305.14463/) (2023)
- [Question Answering Survey: Directions, Challenges, Datasets, Evaluation Matrices](https://mlsanity.com/paper/2112.03572/) (2021)
- [LLM-based Query Expansion Fails for Unfamiliar and Ambiguous Queries](https://mlsanity.com/paper/2505.12694/) (2025)
- [TriviaHG: A Dataset for Automatic Hint Generation from Factoid Questions](https://mlsanity.com/paper/2403.18426/) (2024)
- [Question Difficulty Ranking for Multiple-Choice Reading Comprehension](https://mlsanity.com/paper/2404.10704/) (2024)
- [Are You Doubtful? Oh, It Might Be Difficult Then! Exploring the Use of Model Uncertainty for Question Difficulty Estimation](https://mlsanity.com/paper/2412.11831/) (2024)
- [Wrong Answers Can Also Be Useful: PlausibleQA -- A Large-Scale QA Dataset with Answer Plausibility Scores](https://mlsanity.com/paper/2502.16358/) (2025)
- [WikiHint: A Human-Annotated Dataset for Hint Ranking and Generation](https://mlsanity.com/paper/2412.01626/) (2024)
- [Investigating Entropy for Extractive Document Summarization](https://mlsanity.com/paper/2109.10886/) (2021)
- [Measuring Retrieval Complexity in Question Answering Systems](https://mlsanity.com/paper/2406.03592/) (2024)
- [Exploring the Potential of Large Language Models for Estimating the Reading Comprehension Question Difficulty](https://mlsanity.com/paper/2502.17785/) (2025)

## Built on by (most cited first, 2 of 2)

- [Chronologic: Measuring Language Models' Ability to Represent the Past](https://mlsanity.com/paper/2609.23178/) (2026)
- [What Makes Something Hard(er)? Explaining Question Difficulty in Natural Language](https://mlsanity.com/paper/2610.01627/) (2026)

## Abstract (arXiv)

Estimating question difficulty is a critical component in evaluating and improving large language models (LLMs) for question answering (QA). Existing approaches often rely on readability formulas, retrieval-based signals, or popularity statistics, which may not fully capture the reasoning challenges posed to modern LLMs. In this paper, we introduce Q-DAPS (Question Difficulty based on Answer Plausibility Scores) method, a novel approach that estimates question difficulty by computing the entropy of plausibility scores over candidate answers. We systematically evaluate Q-DAPS across four prominent QA datasets-TriviaQA, NQ, MuSiQue, and QASC-demonstrating that it consistently outperforms baselines. Moreover, Q-DAPS shows strong robustness across hyperparameter variations and question types. Extensive ablation studies further show that Q-DAPS remains robust across different plausibility estimation paradigms, model sizes, and realistic settings. Human evaluations further confirm strong alignment between Q-DAPS's difficulty estimates and human judgments of question difficulty. Overall, Q-DAPS provides an interpretable, scalable, and bias-resilient approach to question difficulty estimation in modern QA systems.