# Wait, that's not an option: LLMs Robustness with Incorrect Multiple-Choice Options

arXiv: https://arxiv.org/abs/2409.00113 · HTML: https://mlsanity.com/paper/2409.00113/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 34 references that are in the map.
- First posted on arXiv 27 Aug 2024.

## Builds on (most cited first, 22 of 22)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset](https://mlsanity.com/paper/2307.04657/) (2023)
- [ORPO: Monolithic Preference Optimization without Reference Model](https://mlsanity.com/paper/2403.07691/) (2024)
- [Large Language Models Are Not Robust Multiple Choice Selectors](https://mlsanity.com/paper/2309.03882/) (2023)
- [Observational Scaling Laws and the Predictability of Language Model Performance](https://mlsanity.com/paper/2405.10938/) (2024)
- [Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arena](https://mlsanity.com/paper/2406.07545/) (2024)
- [Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?](https://mlsanity.com/paper/2402.12483/) (2024)
- [Can multiple-choice questions really be useful in detecting the abilities of LLMs?](https://mlsanity.com/paper/2403.17752/) (2024)
- [Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedback](https://mlsanity.com/paper/2403.18349/) (2024)
- [Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Models](https://mlsanity.com/paper/2407.16221/) (2024)
- [Learn to Refuse: Making Large Language Models More Controllable and Reliable through Knowledge Scope Limitation and Refusal Mechanism](https://mlsanity.com/paper/2311.01041/) (2023)
- [Large Language Models (GPT) Struggle to Answer Multiple-Choice Questions about Code](https://mlsanity.com/paper/2303.08033/) (2023)
- [Multiple-Choice Questions are Efficient and Robust LLM Evaluators](https://mlsanity.com/paper/2405.11966/) (2024)
- [When to Retrieve: Teaching LLMs to Utilize Information Retrieval Effectively](https://mlsanity.com/paper/2404.19705/) (2024)

## Built on by (most cited first, 11 of 11)

- [Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Above](https://mlsanity.com/paper/2502.14127/) (2025)
- [MCP Security Bench (MSB): Benchmarking Attacks Against Model Context Protocol in LLM Agents](https://mlsanity.com/paper/2510.15994/) (2025)
- [Bridging the Knowledge-Prediction Gap in LLMs on Multiple-Choice Questions](https://mlsanity.com/paper/2509.23782/) (2025)
- [Chain-of-Trigger: An Agentic Backdoor that Paradoxically Enhances Agentic Robustness](https://mlsanity.com/paper/2510.08238/) (2025)
- [When No Answer Is Correct: Diagnosing Absent Answer Detection for MLLMs in Video Understanding](https://mlsanity.com/paper/2606.08239/) (2026)
- [Improving Cross-Format Robustness in Language Models with Multi-Format Training](https://mlsanity.com/paper/2606.11643/) (2026)
- [Beyond Multiple Choice: Verifiable OpenQA for Robust Vision-Language RFT](https://mlsanity.com/paper/2511.17405/) (2025)
- [OI-Bench: An Option Injection Benchmark for Evaluating LLM Susceptibility to Directive Interference](https://mlsanity.com/paper/2601.13300/) (2026)
- [Discovering the Hidden Role of Gini Index In Prompt-based Classification](https://mlsanity.com/paper/2603.15654/) (2026)
- [mmPISA-bench: Do LLMs Reason Equally Well Across 43 Languages?](https://mlsanity.com/paper/2606.07069/) (2026)
- [Task Competence Is Not Instruction Following: Evaluating Instruction-Conflicting Behavior in Small Language Models](https://mlsanity.com/paper/2607.19608/) (2026)

## Abstract (arXiv)

This work introduces a novel framework for evaluating LLMs' capacity to balance instruction-following with critical reasoning when presented with multiple-choice questions containing no valid answers. Through systematic evaluation across arithmetic, domain-specific knowledge, and high-stakes medical decision tasks, we demonstrate that post-training aligned models often default to selecting invalid options, while base models exhibit improved refusal capabilities that scale with model size. Our analysis reveals that alignment techniques, though intended to enhance helpfulness, can inadvertently impair models' reflective judgment--the ability to override default behaviors when faced with invalid options. We additionally conduct a parallel human study showing similar instruction-following biases, with implications for how these biases may propagate through human feedback datasets used in alignment. We provide extensive ablation studies examining the impact of model size, training techniques, and prompt engineering. Our findings highlight fundamental tensions between alignment optimization and preservation of critical reasoning capabilities, with important implications for developing more robust AI systems for real-world deployment.