Rethinking Multiple-Choice Questions for RLVR: Unlocking Potential via Distractor Design
On the map: LLM reasoning → Reinforcement learning for reasoning
Abstract (arXiv)
Reinforcement Learning with Verifiable Rewards (RLVR) significantly enhances the reasoning capabilities of Large Language Models. When applied to RLVR, Multiple-Choice Questions (MCQs) offer a scalable source of verifiable data but risk inducing reward hacking, where models shortcut reasoning via random guessing or simple elimination. Current approaches often mitigate this by converting MCQs to open-ended formats, thereby discarding the contrastive signal provided by expert-designed distractors. In this work, we systematically investigate the impact of option design on RLVR. Our analysis highlights two primary insights: (1) Mismatches in option counts between training and testing degrade performance. (2) Strong distractors effectively mitigate random guessing, enabling effective RLVR training even with 2-way questions. Motivated by these findings, we propose Iterative Distractor Curation (IDC), a framework that actively constructs high-quality distractors to block elimination shortcuts and promote deep reasoning. Experiments on various benchmarks demonstrate that our method effectively enhances distractor quality and yields significant gains in RLVR training compared to the original data.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 38 of its 54 references that are in the map. First posted on arXiv 13 Mar 2026.
Builds on
38 of 54 references.
Closest ancestors · 14
Same direction, newest first
- 2025Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination14 Jul
- 2025Spurious Rewards: Rethinking Training Signals in RLVR12 Jun
- 2025From Accuracy to Robustness: A Study of Rule- and Model-based Verifiers in Mathematical Reasoning28 May
- 2025Nemotron-CrossThink: Scaling Self-Learning beyond Math Reasoning15 Apr
- 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
- 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
- 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
- 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning20 Feb
- 2025Process Reinforcement through Implicit Rewards3 Feb
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
- 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
2 more
Other references · 24
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
- 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
- 2016Concrete Problems in AI Safetycited by 1,642
- 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
- 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
- 2025Group Sequence Policy Optimizationcited by 835
- 2025Seed1.5-VL Technical Reportcited by 373
- 2025MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understandingcited by 224
- 2025SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplinescited by 79
- 2024A Survey on Data Synthesis and Augmentation for Large Language Modelscited by 41
12 more
- 2025Med-RLVR: Emerging Medical Reasoning from a 3B base model via reinforcement Learningcited by 40
- 2025Beyond Distillation: Pushing the Limits of Medical LLM Reasoning with Minimalist Rule-Based RLcited by 36
- 2024IOPO: Empowering LLMs with Complex Instruction Following via Input-Output Preference Optimizationcited by 31
- 2025Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluationcited by 30
- 2020Better Distractions: Transformer-based Distractor Generation and Multiple Choice Question Filteringcited by 17
- 2024Exploring Automated Distractor Generation for Math Multiple-choice Questions via Large Language Modelscited by 15
- 2024Improving Automated Distractor Generation for Math Multiple-choice Questions with Overgenerate-and-rankcited by 12
- 2024Long-Span Question-Answering: Automatic Question Generation and QA-System Ranking via Side-by-Side Evaluationcited by 8
- 2025Do LLMs Make Mistakes Like Students? Exploring Natural Alignment between Language Models and Human Error Patternscited by 8
- 2025A Survey of Inductive Reasoning for Large Language Modelscited by 8
- 2024DiVERT: Distractor Generation with Variational Errors Represented as Text for Math Multiple-choice Questionscited by 7
- 2025The Imitation Game for Educational AIcited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Extending RLVR to Open-Ended Tasks via Verifiable Multiple-Choice Reformulation
- 2026Adaptive Ability Decomposing for Unlocking Large Reasoning Model Effective Reinforcement Learning
- 2026Golden Goose: A Simple Trick to Synthesize Unlimited RLVR Tasks from Unverifiable Internet Text
- 2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models
- 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
- 2026JURY-RL: Votes Propose, Proofs Dispose for Label-Free RLVR
- 2026Reinforcement Learning with Verifiable Rewards for Small Search Agents
- 2025RLPR: Extrapolating RLVR to General Domains without Verifiers