# Multiple-Choice Question Generation: Towards an Automated Assessment Framework

arXiv: https://arxiv.org/abs/2209.11830 · HTML: https://mlsanity.com/paper/2209.11830/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 50 references that are in the map.
- First posted on arXiv 23 Sep 2022.

## Builds on (most cited first, 23 of 23)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://mlsanity.com/paper/1704.04683/) (2017)
- [Unified Language Model Pre-training for Natural Language Understanding and Generation](https://mlsanity.com/paper/1905.03197/) (2019)
- [Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning](https://mlsanity.com/paper/1909.00277/) (2019)
- [ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning](https://mlsanity.com/paper/2002.04326/) (2020)
- [Professor Forcing: A New Algorithm for Training Recurrent Networks](https://mlsanity.com/paper/1610.09038/) (2016)
- [UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training](https://mlsanity.com/paper/2002.12804/) (2020)
- [Learning to Ask: Neural Question Generation for Reading Comprehension](https://mlsanity.com/paper/1705.00106/) (2017)
- [Retrospective Reader for Machine Reading Comprehension](https://mlsanity.com/paper/2001.09694/) (2020)
- [A Survey on Machine Reading Comprehension: Tasks, Evaluation Metrics and Benchmark Datasets](https://mlsanity.com/paper/2006.11880/) (2020)
- [Improving Neural Question Generation using Answer Separation](https://mlsanity.com/paper/1809.02393/) (2018)
- [Guiding the Growth: Difficulty-Controllable Question Generation through Step-by-Step Rewriting](https://mlsanity.com/paper/2105.11698/) (2021)
- [Multi-Reference Training with Pseudo-References for Neural Translation and Text Generation](https://mlsanity.com/paper/1808.09564/) (2018)
- [Improving Question Generation With to the Point Context](https://mlsanity.com/paper/1910.06036/) (2019)
- [Question Generation by Transformers](https://mlsanity.com/paper/1909.05017/) (2019)
- [AVA: an Automatic eValuation Approach to Question Answering Systems](https://mlsanity.com/paper/2005.00705/) (2020)

## Built on by (most cited first, 23 of 23)

- [A Survey of Useful LLM Evaluation](https://mlsanity.com/paper/2406.00936/) (2024)
- [On the application of Large Language Models for language teaching and assessment technology](https://mlsanity.com/paper/2307.08393/) (2023)
- [Multimodality of AI for Education: Towards Artificial General Intelligence](https://mlsanity.com/paper/2312.06037/) (2023)
- [Do LLMs Give Psychometrically Plausible Responses in Educational Assessments?](https://mlsanity.com/paper/2506.09796/) (2025)
- [A review on the use of large language models as virtual tutors](https://mlsanity.com/paper/2405.11983/) (2024)
- [AuditLLM: A Tool for Auditing Large Language Models Using Multiprobe Approach](https://mlsanity.com/paper/2402.09334/) (2024)
- [An Automatic Question Usability Evaluation Toolkit](https://mlsanity.com/paper/2405.20529/) (2024)
- [LLMAuditor: A Framework for Auditing Large Language Models Using Human-in-the-Loop](https://mlsanity.com/paper/2402.09346/) (2024)
- [Analysis of the Cambridge Multiple-Choice Questions Reading Dataset with a Focus on Candidate Response Distribution](https://mlsanity.com/paper/2306.13047/) (2023)
- [Unreflected Acceptance -- Investigating the Negative Consequences of ChatGPT-Assisted Problem Solving in Physics Education](https://mlsanity.com/paper/2309.03087/) (2023)
- [Question Difficulty Ranking for Multiple-Choice Reading Comprehension](https://mlsanity.com/paper/2404.10704/) (2024)
- [Dr.Academy: A Benchmark for Evaluating Questioning Capability in Education for Large Language Models](https://mlsanity.com/paper/2408.10947/) (2024)
- [Automatic Generation and Evaluation of Reading Comprehension Test Items with Large Language Models](https://mlsanity.com/paper/2404.07720/) (2024)
- [BenchMarker: An Education-Inspired Toolkit for Highlighting Flaws in Multiple-Choice Benchmarks](https://mlsanity.com/paper/2602.06221/) (2026)
- [Analyzing Multiple-Choice Reading and Listening Comprehension Tests](https://mlsanity.com/paper/2307.01076/) (2023)
- [KNIGHT: Knowledge Graph-Driven Multiple-Choice Question Generation with Adaptive Hardness Calibration](https://mlsanity.com/paper/2602.20135/) (2026)
- [Assessing Distractors in Multiple-Choice Tests](https://mlsanity.com/paper/2311.04554/) (2023)
- [Automated Reading Passage Generation with OpenAI's Large Language Model](https://mlsanity.com/paper/2304.04616/) (2023)
- [Cognitively Diverse Multiple-Choice Question Generation: A Hybrid Multi-Agent Framework with Large Language Models](https://mlsanity.com/paper/2602.03704/) (2026)
- [UniPoll: A Unified Social Media Poll Generation Framework via Multi-Objective Optimization](https://mlsanity.com/paper/2306.06851/) (2023)
- [An Information-Theoretic Approach to Analyze NLP Classification Tasks](https://mlsanity.com/paper/2402.00978/) (2024)
- [EMBRACE: Evaluation and Modifications for Boosting RACE](https://mlsanity.com/paper/2305.08433/) (2023)
- [BengaliMCQ: Automatic Generation and Answer Prediction of Academic Multiple-Choice Questions in a Low-Resource Language](https://mlsanity.com/paper/2608.15547/) (2026)

## Abstract (arXiv)

Automated question generation is an important approach to enable personalisation of English comprehension assessment. Recently, transformer-based pretrained language models have demonstrated the ability to produce appropriate questions from a context paragraph. Typically, these systems are evaluated against a reference set of manually generated questions using n-gram based metrics, or manual qualitative assessment. Here, we focus on a fully automated multiple-choice question generation (MCQG) system where both the question and possible answers must be generated from the context paragraph. Applying n-gram based approaches is challenging for this form of system as the reference set is unlikely to capture the full range of possible questions and answer options. Conversely manual assessment scales poorly and is expensive for MCQG system development. In this work, we propose a set of performance criteria that assess different aspects of the generated multiple-choice questions of interest. These qualities include: grammatical correctness, answerability, diversity and complexity. Initial systems for each of these metrics are described, and individually evaluated on standard multiple-choice reading comprehension corpora.