# Improving Automated Distractor Generation for Math Multiple-choice Questions with Overgenerate-and-rank

arXiv: https://arxiv.org/abs/2405.05144 · HTML: https://mlsanity.com/paper/2405.05144/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 24 references that are in the map.
- First posted on arXiv 19 Apr 2024.

## Builds on (most cited first, 10 of 10)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [Automatic Distractor Generation for Multiple Choice Questions in Standard Tests](https://mlsanity.com/paper/2011.13100/) (2020)
- [Automated Distractor and Feedback Generation for Math Multiple-choice Questions via In-context Learning](https://mlsanity.com/paper/2308.03234/) (2023)
- [Automatic Short Math Answer Grading via In-context Meta-learning](https://mlsanity.com/paper/2205.15219/) (2022)
- [Improving Socratic Question Generation using Data Augmentation and Preference Optimization](https://mlsanity.com/paper/2403.00199/) (2024)
- [Improving Reading Comprehension Question Generation with Data Augmentation and Overgenerate-and-rank](https://mlsanity.com/paper/2306.08847/) (2023)
- [Math Operation Embeddings for Open-ended Solution Analysis and Feedback](https://mlsanity.com/paper/2104.12047/) (2021)

## Built on by (most cited first, 12 of 12)

- [Learning to Make MISTAKEs: Modeling Incorrect Student Thinking And Key Errors](https://mlsanity.com/paper/2510.11502/) (2025)
- [Generating Plausible Distractors for Multiple-Choice Questions via Student Choice Prediction](https://mlsanity.com/paper/2501.13125/) (2025)
- [Do LLMs Make Mistakes Like Students? Exploring Natural Alignment between Language Models and Human Error Patterns](https://mlsanity.com/paper/2502.15140/) (2025)
- [DiVERT: Distractor Generation with Variational Errors Represented as Text for Math Multiple-choice Questions](https://mlsanity.com/paper/2406.19356/) (2024)
- [LookAlike: Consistent Distractor Generation in Math MCQs](https://mlsanity.com/paper/2505.01903/) (2025)
- [Opportunities and Challenges of LLMs in Education: An NLP Perspective](https://mlsanity.com/paper/2507.22753/) (2025)
- [Tailoring Diagnostic Modeling to Individual Learners: Personalized Distractor Generation via MCTS-Guided Reasoning Reconstruction](https://mlsanity.com/paper/2508.11184/) (2025)
- [FEAT: A Preference Feedback Dataset through a Cost-Effective Auto-Generation and Labeling Framework for English AI Tutoring](https://mlsanity.com/paper/2506.19325/) (2025)
- [Prompting Strategies for Language Model-Based Item Generation in K-12 Education: Bridging the Gap Between Small and Large Language Models](https://mlsanity.com/paper/2508.20217/) (2025)
- [Rethinking Multiple-Choice Questions for RLVR: Unlocking Potential via Distractor Design](https://mlsanity.com/paper/2603.12826/) (2026)
- [MAGMA-Edu: Multi-Agent Generative Multimodal Framework for Text-Diagram Educational Question Generation](https://mlsanity.com/paper/2511.18714/) (2025)
- [Incentives Of EdTech: A Systematic Review Of EduNLP Research](https://mlsanity.com/paper/2606.13691/) (2026)

## Abstract (arXiv)

Multiple-choice questions (MCQs) are commonly used across all levels of math education since they can be deployed and graded at a large scale. A critical component of MCQs is the distractors, i.e., incorrect answers crafted to reflect student errors or misconceptions. Automatically generating them in math MCQs, e.g., with large language models, has been challenging. In this work, we propose a novel method to enhance the quality of generated distractors through overgenerate-and-rank, training a ranking model to predict how likely distractors are to be selected by real students. Experimental results on a real-world dataset and human evaluation with math teachers show that our ranking model increases alignment with human-authored distractors, although human-authored ones are still preferred over generated ones.