# Improving the Robustness of QA Models to Challenge Sets with Variational Question-Answer Pair Generation

arXiv: https://arxiv.org/abs/2004.03238 · HTML: https://mlsanity.com/paper/2004.03238/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Apr 2020.

## Built on by (most cited first, 10 of 10)

- [Diverse Text Generation via Variational Encoder-Decoder Models with Gaussian Process Priors](https://mlsanity.com/paper/2204.01227/) (2022)
- [ASSERT: Automated Safety Scenario Red Teaming for Evaluating the Robustness of Large Language Models](https://mlsanity.com/paper/2310.09624/) (2023)
- [Which Shortcut Solution Do Question Answering Models Prefer to Learn?](https://mlsanity.com/paper/2211.16220/) (2022)
- [Can Question Generation Debias Question Answering Models? A Case Study on Question-Context Lexical Overlap](https://mlsanity.com/paper/2109.11256/) (2021)
- [DialogQAE: N-to-N Question Answer Pair Extraction from Customer Service Chatlog](https://mlsanity.com/paper/2212.07112/) (2022)
- [Consecutive Question Generation via Dynamic Multitask Learning](https://mlsanity.com/paper/2211.08850/) (2022)
- [Grounding Synthetic Data Evaluations of Language Models in Unsupervised Document Corpora](https://mlsanity.com/paper/2505.08905/) (2025)
- [Single-Sentence Reader: A Novel Approach for Addressing Answer Position Bias](https://mlsanity.com/paper/2308.04566/) (2023)
- [How to Build Robust FAQ Chatbot with Controllable Question Generator?](https://mlsanity.com/paper/2112.03007/) (2021)
- [An Understanding-Oriented Robust Machine Reading Comprehension Model](https://mlsanity.com/paper/2207.00187/) (2022)

## Abstract (arXiv)

Question answering (QA) models for reading comprehension have achieved human-level accuracy on in-distribution test sets. However, they have been demonstrated to lack robustness to challenge sets, whose distribution is different from that of training sets. Existing data augmentation methods mitigate this problem by simply augmenting training sets with synthetic examples sampled from the same distribution as the challenge sets. However, these methods assume that the distribution of a challenge set is known a priori, making them less applicable to unseen challenge sets. In this study, we focus on question-answer pair generation (QAG) to mitigate this problem. While most existing QAG methods aim to improve the quality of synthetic examples, we conjecture that diversity-promoting QAG can mitigate the sparsity of training sets and lead to better robustness. We present a variational QAG model that generates multiple diverse QA pairs from a paragraph. Our experiments show that our method can improve the accuracy of 12 challenge sets, as well as the in-distribution accuracy. Our code and data are available at https://github.com/KazutoshiShinoda/VQAG.