# Quiz Design Task: Helping Teachers Create Quizzes with Automated Question Generation

arXiv: https://arxiv.org/abs/2205.01730 · HTML: https://mlsanity.com/paper/2205.01730/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 41 references that are in the map.
- First posted on arXiv 3 May 2022.

## Builds on (most cited first, 17 of 17)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [NewsQA: A Machine Comprehension Dataset](https://mlsanity.com/paper/1611.09830/) (2016)
- [Why We Need New Evaluation Metrics for NLG](https://mlsanity.com/paper/1707.06875/) (2017)
- [ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training](https://mlsanity.com/paper/2001.04063/) (2020)
- [Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation](https://mlsanity.com/paper/1703.09902/) (2017)
- [Learning to Ask: Neural Question Generation for Reading Comprehension](https://mlsanity.com/paper/1705.00106/) (2017)
- [Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information](https://mlsanity.com/paper/1805.04655/) (2018)
- [The price of debiasing automatic metrics in natural language evaluation](https://mlsanity.com/paper/1807.02202/) (2018)
- [Towards a Better Metric for Evaluating Question Generation Systems](https://mlsanity.com/paper/1808.10192/) (2018)
- [MixQG: Neural Question Generation with Mixed Answer Types](https://mlsanity.com/paper/2110.08175/) (2021)
- [Evaluating Rewards for Question Generation Models](https://mlsanity.com/paper/1902.11049/) (2019)
- [What's The Latest? A Question-driven News Chatbot](https://mlsanity.com/paper/2105.05392/) (2021)
- [I Do Not Understand What I Cannot Define: Automatic Question Generation With Pedagogically-Driven Content Selection](https://mlsanity.com/paper/2110.04123/) (2021)

## Built on by (most cited first, 12 of 12)

- [Leveraging Large Language Models for Learning Complex Legal Concepts through Storytelling](https://mlsanity.com/paper/2402.17019/) (2024)
- [RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Question](https://mlsanity.com/paper/2211.01482/) (2022)
- [How Teachers Can Use Large Language Models and Bloom's Taxonomy to Create Educational Quizzes](https://mlsanity.com/paper/2401.05914/) (2024)
- [QGEval: Benchmarking Multi-dimensional Evaluation for Question Generation](https://mlsanity.com/paper/2406.05707/) (2024)
- [FOLLOWUPQG: Towards Information-Seeking Follow-up Question Generation](https://mlsanity.com/paper/2309.05007/) (2023)
- [Evaluating the Knowledge Dependency of Questions](https://mlsanity.com/paper/2211.11902/) (2022)
- [Evaluation of Question Generation Needs More References](https://mlsanity.com/paper/2305.16626/) (2023)
- [CONSISTENT: Open-Ended Question Generation From News Articles](https://mlsanity.com/paper/2210.11536/) (2022)
- [Reference-based Metrics Disprove Themselves in Question Generation](https://mlsanity.com/paper/2403.12242/) (2024)
- [Bridging Information Gaps with Comprehensive Answers: Improving the Diversity and Informativeness of Follow-Up Questions](https://mlsanity.com/paper/2502.17715/) (2025)
- [Near-Negative Distinction: Giving a Second Life to Human Evaluation Datasets](https://mlsanity.com/paper/2205.06871/) (2022)
- [From Memorization to Creation: Evaluating the Cognitive Depth of LLM-Generated Educational Questions](https://mlsanity.com/paper/2606.18257/) (2026)

## Abstract (arXiv)

Question generation (QGen) models are often evaluated with standardized NLG metrics that are based on n-gram overlap. In this paper, we measure whether these metric improvements translate to gains in a practical setting, focusing on the use case of helping teachers automate the generation of reading comprehension quizzes. In our study, teachers building a quiz receive question suggestions, which they can either accept or refuse with a reason. Even though we find that recent progress in QGen leads to a significant increase in question acceptance rates, there is still large room for improvement, with the best model having only 68.4% of its questions accepted by the ten teachers who participated in our study. We then leverage the annotations we collected to analyze standard NLG metrics and find that model performance has reached projected upper-bounds, suggesting new automatic metrics are needed to guide QGen research forward.