# Can Question Generation Debias Question Answering Models? A Case Study on Question-Context Lexical Overlap

arXiv: https://arxiv.org/abs/2109.11256 · HTML: https://mlsanity.com/paper/2109.11256/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 35 references that are in the map.
- First posted on arXiv 23 Sep 2021.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://mlsanity.com/paper/1804.06876/) (2018)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [Incorporating Copying Mechanism in Sequence-to-Sequence Learning](https://mlsanity.com/paper/1603.06393/) (2016)
- [Learning to Ask: Neural Question Generation for Reading Comprehension](https://mlsanity.com/paper/1705.00106/) (2017)
- [Look at the First Sentence: Position Bias in Question Answering](https://mlsanity.com/paper/2004.14602/) (2020)
- [Learning and Evaluating General Linguistic Intelligence](https://mlsanity.com/paper/1901.11373/) (2019)
- [Neural Question Generation from Text: A Preliminary Study](https://mlsanity.com/paper/1704.01792/) (2017)
- [Understanding data augmentation for classification: when to warp?](https://mlsanity.com/paper/1609.08764/) (2016)
- [Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering](https://mlsanity.com/paper/1909.06356/) (2019)
- [What Makes Reading Comprehension Questions Easier?](https://mlsanity.com/paper/1808.09384/) (2018)
- [MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension](https://mlsanity.com/paper/1905.13453/) (2019)
- [Harvesting Paragraph-Level Question-Answer Pairs from Wikipedia](https://mlsanity.com/paper/1805.05942/) (2018)
- [What do Models Learn from Question Answering Datasets?](https://mlsanity.com/paper/2004.03490/) (2020)
- [Pointing the Unknown Words](https://mlsanity.com/paper/1603.08148/) (2016)
- [On The Evaluation of Machine Translation Systems Trained With Back-Translation](https://mlsanity.com/paper/1908.05204/) (2019)
- [Towards Robustifying NLI Models Against Lexical Dataset Biases](https://mlsanity.com/paper/2005.04732/) (2020)
- [Generating Diverse and Consistent QA pairs from Contexts with Information-Maximizing Hierarchical Conditional VAEs](https://mlsanity.com/paper/2005.13837/) (2020)
- [Semi-Supervised QA with Generative Domain-Adaptive Nets](https://mlsanity.com/paper/1702.02206/) (2017)
- [Simple and Effective Semi-Supervised Question Answering](https://mlsanity.com/paper/1804.00720/) (2018)
- [Improving the Robustness of QA Models to Challenge Sets with Variational Question-Answer Pair Generation](https://mlsanity.com/paper/2004.03238/) (2020)

## Built on by (most cited first, 8 of 8)

- [Hyperlink-induced Pre-training for Passage Retrieval in Open-domain Question Answering](https://mlsanity.com/paper/2203.06942/) (2022)
- [Low-Resource Dense Retrieval for Open-Domain Question Answering: A Comprehensive Survey](https://mlsanity.com/paper/2208.03197/) (2022)
- [Which Shortcut Solution Do Question Answering Models Prefer to Learn?](https://mlsanity.com/paper/2211.16220/) (2022)
- [Look to the Right: Mitigating Relative Position Bias in Extractive Question Answering](https://mlsanity.com/paper/2210.14541/) (2022)
- [Think Twice: Measuring the Efficiency of Eliminating Prediction Shortcuts of Question Answering Models](https://mlsanity.com/paper/2305.06841/) (2023)
- [XAIQA: Explainer-Based Data Augmentation for Extractive Question Answering](https://mlsanity.com/paper/2312.03567/) (2023)
- [Can Out-of-Distribution Evaluations Uncover Reliance on Shortcuts? A Case Study in Question Answering](https://mlsanity.com/paper/2508.18407/) (2025)
- [Debiasing Reward Models via Causally Motivated Inference-Time Intervention](https://mlsanity.com/paper/2604.27495/) (2026)

## Abstract (arXiv)

Question answering (QA) models for reading comprehension have been demonstrated to exploit unintended dataset biases such as question-context lexical overlap. This hinders QA models from generalizing to under-represented samples such as questions with low lexical overlap. Question generation (QG), a method for augmenting QA datasets, can be a solution for such performance degradation if QG can properly debias QA datasets. However, we discover that recent neural QG models are biased towards generating questions with high lexical overlap, which can amplify the dataset bias. Moreover, our analysis reveals that data augmentation with these QG models frequently impairs the performance on questions with low lexical overlap, while improving that on questions with high lexical overlap. To address this problem, we use a synonym replacement-based approach to augment questions with low lexical overlap. We demonstrate that the proposed data augmentation approach is simple yet effective to mitigate the degradation problem with only 70k synthetic examples. Our data is publicly available at https://github.com/KazutoshiShinoda/Synonym-Replacement.