# Does Multiple Choice Have a Future in the Age of Generative AI? A Posttest-only RCT

arXiv: https://arxiv.org/abs/2412.10267 · HTML: https://mlsanity.com/paper/2412.10267/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 43 references that are in the map.
- First posted on arXiv 13 Dec 2024.

## Builds on (most cited first, 9 of 9)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Applying Large Language Models and Chain-of-Thought for Automatic Scoring](https://mlsanity.com/paper/2312.03748/) (2023)
- [How Can I Get It Right? Using GPT to Rephrase Incorrect Trainee Responses](https://mlsanity.com/paper/2405.00970/) (2024)
- [Can Large Language Models Make the Grade? An Empirical Study Evaluating LLMs Ability to Mark Short Answer Questions in K-12 Education](https://mlsanity.com/paper/2405.02985/) (2024)
- [How Can I Improve? Using GPT to Highlight the Desired and Undesired Parts of Open-ended Responses](https://mlsanity.com/paper/2405.00291/) (2024)
- [Using Large Language Models to Assess Tutors' Performance in Reacting to Students Making Math Errors](https://mlsanity.com/paper/2401.03238/) (2024)
- [Comparative Analysis of GPT-4 and Human Graders in Evaluating Praise Given to Students in Synthetic Dialogues](https://mlsanity.com/paper/2307.02018/) (2023)

## Built on by (most cited first, 7 of 7)

- [Reasoning and Sampling-Augmented MCQ Difficulty Prediction via LLMs](https://mlsanity.com/paper/2503.08551/) (2025)
- [LookAlike: Consistent Distractor Generation in Math MCQs](https://mlsanity.com/paper/2505.01903/) (2025)
- [Augmenting Human-Annotated Training Data with Large Language Model Generation and Distillation in Open-Response Assessment](https://mlsanity.com/paper/2501.09126/) (2025)
- [Tracing the Data Trail: A Survey of Data Provenance, Transparency and Traceability in LLMs](https://mlsanity.com/paper/2601.14311/) (2026)
- [Beyond Agreement: Rethinking Ground Truth in Educational AI Annotation](https://mlsanity.com/paper/2508.00143/) (2025)
- [Harnessing Structured Knowledge: A Concept Map-Based Approach for High-Quality Multiple Choice Question Generation with Effective Distractors](https://mlsanity.com/paper/2505.02850/) (2025)
- [AI-Driven Assessment of Human Tutors: Linking Training Performance to Real-Life Practice](https://mlsanity.com/paper/2606.18617/) (2026)

## Abstract (arXiv)

The role of multiple-choice questions (MCQs) as effective learning tools has been debated in past research. While MCQs are widely used due to their ease in grading, open response questions are increasingly used for instruction, given advances in large language models (LLMs) for automated grading. This study evaluates MCQs effectiveness relative to open-response questions, both individually and in combination, on learning. These activities are embedded within six tutor lessons on advocacy. Using a posttest-only randomized control design, we compare the performance of 234 tutors (790 lesson completions) across three conditions: MCQ only, open response only, and a combination of both. We find no significant learning differences across conditions at posttest, but tutors in the MCQ condition took significantly less time to complete instruction. These findings suggest that MCQs are as effective, and more efficient, than open response tasks for learning when practice time is limited. To further enhance efficiency, we autograded open responses using GPT-4o and GPT-4-turbo. GPT models demonstrate proficiency for purposes of low-stakes assessment, though further research is needed for broader use. This study contributes a dataset of lesson log data, human annotation rubrics, and LLM prompts to promote transparency and reproducibility.