# Evaluating Rewards for Question Generation Models

arXiv: https://arxiv.org/abs/1902.11049 · HTML: https://mlsanity.com/paper/1902.11049/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Feb 2019.

## Built on by (most cited first, 15 of 15)

- [PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them](https://mlsanity.com/paper/2102.07033/) (2021)
- [Varifocal Question Generation for Fact-checking](https://mlsanity.com/paper/2210.12400/) (2022)
- [On the application of Large Language Models for language teaching and assessment technology](https://mlsanity.com/paper/2307.08393/) (2023)
- [ProQA: Structural Prompt-based Pre-training for Unified Question Answering](https://mlsanity.com/paper/2205.04040/) (2022)
- [RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Question](https://mlsanity.com/paper/2211.01482/) (2022)
- [Quiz Design Task: Helping Teachers Create Quizzes with Automated Question Generation](https://mlsanity.com/paper/2205.01730/) (2022)
- [Diversify Question Generation with Retrieval-Augmented Style Transfer](https://mlsanity.com/paper/2310.14503/) (2023)
- [Evaluating the Knowledge Dependency of Questions](https://mlsanity.com/paper/2211.11902/) (2022)
- [Asking Questions Like Educational Experts: Automatically Generating Question-Answer Pairs on Real-World Examination Data](https://mlsanity.com/paper/2109.05179/) (2021)
- [Generating Self-Contained and Summary-Centric Question Answer Pairs via Differentiable Reward Imitation Learning](https://mlsanity.com/paper/2109.04689/) (2021)
- [A Pipeline for Generating, Annotating and Employing Synthetic Data for Real World Question Answering](https://mlsanity.com/paper/2211.16971/) (2022)
- [Question Answering with Deep Neural Networks for Semi-Structured Heterogeneous Genealogical Knowledge Graphs](https://mlsanity.com/paper/2307.16214/) (2023)
- [Ask To The Point: Open-Domain Entity-Centric Question Generation](https://mlsanity.com/paper/2310.14126/) (2023)
- [ECIS-VQG: Generation of Entity-centric Information-seeking Questions from Videos](https://mlsanity.com/paper/2410.09776/) (2024)
- [Automatic question generation based on sentence structure analysis using machine learning approach](https://mlsanity.com/paper/2205.12811/) (2022)

## Abstract (arXiv)

Recent approaches to question generation have used modifications to a Seq2Seq architecture inspired by advances in machine translation. Models are trained using teacher forcing to optimise only the one-step-ahead prediction. However, at test time, the model is asked to generate a whole sequence, causing errors to propagate through the generation process (exposure bias). A number of authors have proposed countering this bias by optimising for a reward that is less tightly coupled to the training data, using reinforcement learning. We optimise directly for quality metrics, including a novel approach using a discriminator learned directly from the training data. We confirm that policy gradient methods can be used to decouple training from the ground truth, leading to increases in the metrics used as rewards. We perform a human evaluation, and show that although these metrics have previously been assumed to be good proxies for question quality, they are poorly aligned with human judgement and the model simply learns to exploit the weaknesses of the reward source.