# Distractor Generation in Multiple-Choice Tasks: A Survey of Methods, Datasets, and Evaluation

arXiv: https://arxiv.org/abs/2402.01512 · HTML: https://mlsanity.com/paper/2402.01512/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 74 of its 155 references that are in the map.
- First posted on arXiv 2 Feb 2024.

## Builds on (most cited first, 30 of 74)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://mlsanity.com/paper/1704.04683/) (2017)
- [Crowdsourcing Multiple Choice Science Questions](https://mlsanity.com/paper/1707.06209/) (2017)
- [MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms](https://mlsanity.com/paper/1905.13319/) (2019)
- [Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problems](https://mlsanity.com/paper/1705.04146/) (2017)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning](https://mlsanity.com/paper/2105.04165/) (2021)
- [Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning](https://mlsanity.com/paper/2209.14610/) (2022)
- [DVQA: Understanding Data Visualizations via Question Answering](https://mlsanity.com/paper/1801.08163/) (2018)
- [Visual7W: Grounded Question Answering in Images](https://mlsanity.com/paper/1511.03416/) (2015)
- [MovieQA: Understanding Stories in Movies through Question-Answering](https://mlsanity.com/paper/1512.02902/) (2015)

## Built on by (most cited first, 25 of 25)

- [CausalVQA: A Physically Grounded Causal Reasoning Benchmark for Video Models](https://mlsanity.com/paper/2506.09943/) (2025)
- [MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Research](https://mlsanity.com/paper/2503.13399/) (2025)
- [DiVERT: Distractor Generation with Variational Errors Represented as Text for Math Multiple-choice Questions](https://mlsanity.com/paper/2406.19356/) (2024)
- [LookAlike: Consistent Distractor Generation in Math MCQs](https://mlsanity.com/paper/2505.01903/) (2025)
- [Omni-Modal Dissonance Benchmark: Systematically Breaking Modality Consensus to Probe Robustness and Calibrated Abstention](https://mlsanity.com/paper/2603.27187/) (2026)
- [Wrong Answers Can Also Be Useful: PlausibleQA -- A Large-Scale QA Dataset with Answer Plausibility Scores](https://mlsanity.com/paper/2502.16358/) (2025)
- [Tailoring Diagnostic Modeling to Individual Learners: Personalized Distractor Generation via MCTS-Guided Reasoning Reconstruction](https://mlsanity.com/paper/2508.11184/) (2025)
- [Better Datasets Start From RefineLab: Automatic Optimization for High-Quality Dataset Refinement](https://mlsanity.com/paper/2511.06530/) (2025)
- [A Survey of Pun Generation: Datasets, Evaluations and Methodologies](https://mlsanity.com/paper/2507.04793/) (2025)
- [Question Difficulty Estimation for Large Language Models via Answer Plausibility Scoring](https://mlsanity.com/paper/2605.12398/) (2026)
- [D-GEN: Automatic Distractor Generation and Evaluation for Reliable Assessment of Generative Model](https://mlsanity.com/paper/2504.13439/) (2025)
- [Enhancing Clinical Multiple-Choice Questions Benchmarks with Knowledge Graph Guided Distractor Generation](https://mlsanity.com/paper/2506.00612/) (2025)
- [Difficulty-Controllable Multiple-Choice Question Generation Using Large Language Models and Direct Preference Optimization](https://mlsanity.com/paper/2510.19265/) (2025)
- [MicroVQA++: High-Quality Microscopy Reasoning Dataset with Weakly Supervised Graphs for Multimodal Large Language Model](https://mlsanity.com/paper/2511.11407/) (2025)
- [Reducing Text Bias in Synthetically Generated MCQAs for VLMs in Autonomous Driving](https://mlsanity.com/paper/2602.17677/) (2026)
- [Harnessing Structured Knowledge: A Concept Map-Based Approach for High-Quality Multiple Choice Question Generation with Effective Distractors](https://mlsanity.com/paper/2505.02850/) (2025)
- [From Model to Classroom: Evaluating Generated MCQs for Portuguese with Narrative and Difficulty Concerns](https://mlsanity.com/paper/2506.15598/) (2025)
- [DualReward: A Dynamic Reinforcement Learning Framework for Cloze Tests Distractor Generation](https://mlsanity.com/paper/2507.11875/) (2025)
- [PerCoR: Evaluating Commonsense Reasoning in Persian via Multiple-Choice Sentence Completion](https://mlsanity.com/paper/2510.22616/) (2025)
- [Seeing isn't Hearing: Benchmarking Vision Language Models at Interpreting Spectrograms](https://mlsanity.com/paper/2511.13225/) (2025)
- [Corpus Prevalence of Multiple-Choice Question Options](https://mlsanity.com/paper/2602.17377/) (2026)
- [BAGEL: Benchmarking Animal Knowledge Expertise in Language Models](https://mlsanity.com/paper/2604.16241/) (2026)
- [You Really Didn't Get That? Benchmarking Social Pragmatic Inference for Indirect and Playful Chinese Online Comments](https://mlsanity.com/paper/2609.04384/) (2026)
- [Turkish MMLU Pro: Traceable Option Augmentation and Its Validity Limits in Turkish Multiple-Choice Evaluation](https://mlsanity.com/paper/2609.15467/) (2026)
- [The Architect, the Adversary, and the Judge: Closed-Loop Generation of Standards-Aligned Assessment Items at Scale](https://mlsanity.com/paper/2609.26087/) (2026)

## Abstract (arXiv)

The distractor generation task focuses on generating incorrect but plausible options for objective questions such as fill-in-the-blank and multiple-choice questions. This task is widely utilized in educational settings across various domains and subjects. The effectiveness of these questions in assessments relies on the quality of the distractors, as they challenge examinees to select the correct answer from a set of misleading options. The evolution of artificial intelligence (AI) has transitioned the task from traditional methods to the use of neural networks and pre-trained language models. This shift has established new benchmarks and expanded the use of advanced deep learning methods in generating distractors. This survey explores distractor generation tasks, datasets, methods, and current evaluation metrics for English objective questions, covering both text-based and multi-modal domains. It also evaluates existing AI models and benchmarks and discusses potential future research directions.