# Automated Distractor and Feedback Generation for Math Multiple-choice Questions via In-context Learning

arXiv: https://arxiv.org/abs/2308.03234 · HTML: https://mlsanity.com/paper/2308.03234/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 40 references that are in the map.
- First posted on arXiv 7 Aug 2023.

## Builds on (most cited first, 11 of 11)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [Automatic Distractor Generation for Multiple Choice Questions in Standard Tests](https://mlsanity.com/paper/2011.13100/) (2020)
- [Generating Distractors for Reading Comprehension Questions from Real Examinations](https://mlsanity.com/paper/1809.02768/) (2018)
- [Mathematical Language Processing: Automatic Grading and Feedback for Open Response Mathematical Questions](https://mlsanity.com/paper/1501.04346/) (2015)
- [Automated Personalized Feedback Improves Learning Gains in an Intelligent Tutoring System](https://mlsanity.com/paper/2005.02431/) (2020)
- [BERT-based distractor generation for Swedish reading comprehension questions using a small-scale dataset](https://mlsanity.com/paper/2108.03973/) (2021)
- [Math Operation Embeddings for Open-ended Solution Analysis and Feedback](https://mlsanity.com/paper/2104.12047/) (2021)
- [Feedback Generation for Performance Problems in Introductory Programming Assignments](https://mlsanity.com/paper/1403.4064/) (2014)

## Built on by (most cited first, 24 of 24)

- [Distractor Generation in Multiple-Choice Tasks: A Survey of Methods, Datasets, and Evaluation](https://mlsanity.com/paper/2402.01512/) (2024)
- [Improving the Validity of Automatically Generated Feedback via Reinforcement Learning](https://mlsanity.com/paper/2403.01304/) (2024)
- [Open Source Language Models Can Provide Feedback: Evaluating LLMs' Ability to Help Students Using GPT-4-As-A-Judge](https://mlsanity.com/paper/2405.05253/) (2024)
- [Learning to Make MISTAKEs: Modeling Incorrect Student Thinking And Key Errors](https://mlsanity.com/paper/2510.11502/) (2025)
- [Generative AI for Education (GAIED): Advances, Opportunities, and Challenges](https://mlsanity.com/paper/2402.01580/) (2024)
- [How Can I Get It Right? Using GPT to Rephrase Incorrect Trainee Responses](https://mlsanity.com/paper/2405.00970/) (2024)
- [Improving Automated Distractor Generation for Math Multiple-choice Questions with Overgenerate-and-rank](https://mlsanity.com/paper/2405.05144/) (2024)
- [How Can I Improve? Using GPT to Highlight the Desired and Undesired Parts of Open-ended Responses](https://mlsanity.com/paper/2405.00291/) (2024)
- [Automatic Generation of Question Hints for Mathematics Problems using Large Language Models in Educational Technology](https://mlsanity.com/paper/2411.03495/) (2024)
- [Do LLMs Make Mistakes Like Students? Exploring Natural Alignment between Language Models and Human Error Patterns](https://mlsanity.com/paper/2502.15140/) (2025)
- [BenchMarker: An Education-Inspired Toolkit for Highlighting Flaws in Multiple-Choice Benchmarks](https://mlsanity.com/paper/2602.06221/) (2026)
- [Towards Generalizable Agents in Text-Based Educational Environments: A Study of Integrating RL with LLMs](https://mlsanity.com/paper/2404.18978/) (2024)
- [Can Large Language Models Replicate ITS Feedback on Open-Ended Math Questions?](https://mlsanity.com/paper/2405.06414/) (2024)
- [WikiHint: A Human-Annotated Dataset for Hint Ranking and Generation](https://mlsanity.com/paper/2412.01626/) (2024)
- [Modeling Complex Mathematical Reasoning via Large Language Model based MathAgent](https://mlsanity.com/paper/2312.08926/) (2023)
- [Look Before You Leap: Problem Elaboration Prompting Improves Mathematical Reasoning in Large Language Models](https://mlsanity.com/paper/2402.15764/) (2024)
- [Enhancing Student Learning with LLM-Generated Retrieval Practice Questions: An Empirical Study in Data Science Courses](https://mlsanity.com/paper/2507.05629/) (2025)
- [Leveraging Large Language Models to Generate Course-specific Semantically Annotated Learning Objects](https://mlsanity.com/paper/2412.04185/) (2024)
- [Using language models in the implicit automated assessment of mathematical short answer items](https://mlsanity.com/paper/2308.11006/) (2023)
- [Improving Bilingual Capabilities of Language Models to Support Diverse Linguistic Practices in Education](https://mlsanity.com/paper/2411.04308/) (2024)
- [Harnessing Structured Knowledge: A Concept Map-Based Approach for High-Quality Multiple Choice Question Generation with Effective Distractors](https://mlsanity.com/paper/2505.02850/) (2025)
- [Corpus Prevalence of Multiple-Choice Question Options](https://mlsanity.com/paper/2602.17377/) (2026)
- [Beyond Fine-Tuning: In-Context Learning and Chain-of-Thought for Reasoned Distractor Generation](https://mlsanity.com/paper/2604.17574/) (2026)
- [Towards Pedagogically Aligned LLM Tutors for Math Mistake Remediation](https://mlsanity.com/paper/2606.21502/) (2026)

## Abstract (arXiv)

Multiple-choice questions (MCQs) are ubiquitous in almost all levels of education since they are easy to administer, grade, and are a reliable form of assessment. An important aspect of MCQs is the distractors, i.e., incorrect options that are designed to target specific misconceptions or insufficient knowledge among students. To date, the task of crafting high-quality distractors has largely remained a labor-intensive process for teachers and learning content designers, which has limited scalability. In this work, we explore the task of automated distractor and corresponding feedback message generation in math MCQs using large language models. We establish a formulation of these two tasks and propose a simple, in-context learning-based solution. Moreover, we propose generative AI-based metrics for evaluating the quality of the feedback messages. We conduct extensive experiments on these tasks using a real-world MCQ dataset. Our findings suggest that there is a lot of room for improvement in automated distractor and feedback generation; based on these findings, we outline several directions for future work.