# Plausibly Problematic Questions in Multiple-Choice Benchmarks for Commonsense Reasoning

arXiv: https://arxiv.org/abs/2410.10854 · HTML: https://mlsanity.com/paper/2410.10854/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 6 of its 38 references that are in the map.
- First posted on arXiv 6 Oct 2024.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 6 of 6)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024)
- [Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations](https://mlsanity.com/paper/2205.11822/) (2022)
- [Rissanen Data Analysis: Examining Dataset Characteristics via Description Length](https://mlsanity.com/paper/2103.03872/) (2021)

## Built on by (most cited first, 13 of 13)

- [Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Above](https://mlsanity.com/paper/2502.14127/) (2025)
- [BLAB: Brutally Long Audio Bench](https://mlsanity.com/paper/2505.03054/) (2025)
- [Susu Box or Piggy Bank: Assessing Cultural Commonsense Knowledge between Ghana and the U.S](https://mlsanity.com/paper/2410.16451/) (2024)
- [Garbage In, Reasoning Out? Why Benchmark Scores are Unreliable and What to Do About It](https://mlsanity.com/paper/2506.23864/) (2025)
- [BenchMarker: An Education-Inspired Toolkit for Highlighting Flaws in Multiple-Choice Benchmarks](https://mlsanity.com/paper/2602.06221/) (2026)
- [A large-scale evaluation of commonsense knowledge in humans and large language models](https://mlsanity.com/paper/2505.10309/) (2025)
- [Social Story Frames: Contextual Reasoning about Narrative Intent and Reception](https://mlsanity.com/paper/2512.15925/) (2025)
- [Arguments that Alter Minds: LLM Rationales Sway Human (and LLM) Notions of Plausibility](https://mlsanity.com/paper/2510.08091/) (2025)
- [Speaking the Right Language: The Impact of Expertise Alignment in User-AI Interactions](https://mlsanity.com/paper/2502.18685/) (2025)
- [The Dog the Cat Chased Stumped the Model: Measuring When Language Models Abandon Structure for Shortcuts](https://mlsanity.com/paper/2510.20543/) (2025)
- [BAGEL: Benchmarking Animal Knowledge Expertise in Language Models](https://mlsanity.com/paper/2604.16241/) (2026)
- [Benchmarking the Benchmarks: Testing the Predictive Validity of Commonsense Benchmarks](https://mlsanity.com/paper/2608.03340/) (2026)
- [Check The Scoreboard: An Analysis of Scoring Schemes on Multiple-Choice Evaluation](https://mlsanity.com/paper/2608.29887/) (2026)

## Abstract (arXiv)

Questions involving commonsense reasoning about everyday situations often admit many textitpossible or textitplausible answers. In contrast, multiple-choice question (MCQ) benchmarks for commonsense reasoning require a hard selection of a single correct answer, which, in principle, should represent the textitmost plausible answer choice. On 250 MCQ items sampled from two commonsense reasoning benchmarks, we collect 5,000 independent plausibility judgments on answer choices. We find that for over 20% of the sampled MCQs, the answer choice rated most plausible does not match the benchmark gold answers; upon manual inspection, we confirm that this subset exhibits higher rates of problems like ambiguity or semantic mismatch between question and answer choices. Experiments with LLMs reveal low accuracy and high variation in performance on the subset, suggesting our plausibility criterion may be helpful in identifying more reliable benchmark items for commonsense evaluation.