# Wrong Answers Can Also Be Useful: PlausibleQA -- A Large-Scale QA Dataset with Answer Plausibility Scores

arXiv: https://arxiv.org/abs/2502.16358 · HTML: https://mlsanity.com/paper/2502.16358/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 62 references that are in the map.
- First posted on arXiv 22 Feb 2025.

## Builds on (most cited first, 28 of 28)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://mlsanity.com/paper/1704.04683/) (2017)
- [Crowdsourcing Multiple Choice Science Questions](https://mlsanity.com/paper/1707.06209/) (2017)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [Learning the Difference that Makes a Difference with Counterfactually-Augmented Data](https://mlsanity.com/paper/1909.12434/) (2019)
- [Evaluating Open-Domain Question Answering in the Era of Large Language Models](https://mlsanity.com/paper/2305.06984/) (2023)
- [Large Language Models Meet NLP: A Survey](https://mlsanity.com/paper/2405.12819/) (2024)
- [Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation](https://mlsanity.com/paper/2202.07654/) (2022)
- [Multi-hop Question Answering](https://mlsanity.com/paper/2204.09140/) (2022)
- [SPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrieval](https://mlsanity.com/paper/2009.13013/) (2020)
- [Generating Factoid Questions With Recurrent Neural Networks: The 30M Factoid Question-Answer Corpus](https://mlsanity.com/paper/1603.06807/) (2016)
- [Exploring the Impact of Negative Samples of Contrastive Learning: A Case Study of Sentence Embedding](https://mlsanity.com/paper/2202.13093/) (2022)
- [Distractor Generation in Multiple-Choice Tasks: A Survey of Methods, Datasets, and Evaluation](https://mlsanity.com/paper/2402.01512/) (2024)
- [Unsupervised Distractor Generation via Large Language Model Distilling and Counterfactual Contrastive Decoding](https://mlsanity.com/paper/2406.01306/) (2024)
- [Enhancing Distractor Generation for Multiple-Choice Questions with Retrieval Augmented Pretraining and Knowledge Graph Integration](https://mlsanity.com/paper/2406.13578/) (2024)
- [TriviaHG: A Dataset for Automatic Hint Generation from Factoid Questions](https://mlsanity.com/paper/2403.18426/) (2024)
- [Exploring Hint Generation Approaches in Open-Domain Question Answering](https://mlsanity.com/paper/2409.16096/) (2024)
- [Assessing Distractors in Multiple-Choice Tests](https://mlsanity.com/paper/2311.04554/) (2023)
- [Measuring Retrieval Complexity in Question Answering Systems](https://mlsanity.com/paper/2406.03592/) (2024)

## Built on by (most cited first, 5 of 5)

- [Decomposing Factual Sycophancy in Language Models: How Size and Instruction Tuning Shape Robustness](https://mlsanity.com/paper/2606.06306/) (2026)
- [One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models](https://mlsanity.com/paper/2603.03291/) (2026)
- [Question Difficulty Estimation for Large Language Models via Answer Plausibility Scoring](https://mlsanity.com/paper/2605.12398/) (2026)
- [DialDefer: A Framework for Detecting and Mitigating LLM Dialogic Deference](https://mlsanity.com/paper/2601.10896/) (2026)
- [Context Convergence Improves Answering Inferential Questions](https://mlsanity.com/paper/2605.12370/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are revolutionizing information retrieval, with chatbots becoming an important source for answering user queries. As by their design, LLMs prioritize generating correct answers, the value of highly plausible yet incorrect answers (candidate answers) tends to be overlooked. However, such answers can still prove useful, for example, they can play a crucial role in tasks like Multiple-Choice Question Answering (MCQA) and QA Robustness Assessment (QARA). Existing QA datasets primarily focus on correct answers without explicit consideration of the plausibility of other candidate answers, limiting opportunity for more nuanced evaluations of models. To address this gap, we introduce PlausibleQA, a large-scale dataset comprising 10,000 questions and 100,000 candidate answers, each annotated with plausibility scores and justifications for their selection. Additionally, the dataset includes 900,000 justifications for pairwise comparisons between candidate answers, further refining plausibility assessments. We evaluate PlausibleQA through human assessments and empirical experiments, demonstrating its utility in MCQA and QARA analysis. Our findings show that plausibility-aware approaches are effective for MCQA distractor generation and QARA. We release PlausibleQA as a resource for advancing QA research and enhancing LLM performance in distinguishing plausible distractors from correct answers.