# Training Question Answering Models From Synthetic Data

arXiv: https://arxiv.org/abs/2002.09599 · HTML: https://mlsanity.com/paper/2002.09599/

## Key facts

- Cited by 95 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Feb 2020.

## Built on by (most cited first, 30 of 95)

- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models](https://mlsanity.com/paper/2501.09686/) (2025)
- [ZeroGen: Efficient Zero-shot Learning via Dataset Generation](https://mlsanity.com/paper/2202.07922/) (2022)
- [Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation](https://mlsanity.com/paper/2104.08678/) (2021)
- [ProGen: Progressive Zero-shot Dataset Generation via In-context Feedback](https://mlsanity.com/paper/2210.12329/) (2022)
- [All You May Need for VQA are Image Captions](https://mlsanity.com/paper/2205.01883/) (2022)
- [Leveraging Large Language Models in Conversational Recommender Systems](https://mlsanity.com/paper/2305.07961/) (2023)
- [Few-Shot Question Answering by Pretraining Span Selection](https://mlsanity.com/paper/2101.00438/) (2021)
- [When Search Engine Services meet Large Language Models: Visions and Challenges](https://mlsanity.com/paper/2407.00128/) (2024)
- [Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learning](https://mlsanity.com/paper/2205.12679/) (2022)
- [Automatic Generation of Socratic Subquestions for Teaching Math Word Problems](https://mlsanity.com/paper/2211.12835/) (2022)
- [Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills](https://mlsanity.com/paper/2107.07261/) (2021)
- [STraTA: Self-Training with Task Augmentation for Better Few-shot Learning](https://mlsanity.com/paper/2109.06270/) (2021)
- [Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistants](https://mlsanity.com/paper/2112.09062/) (2021)
- [ArchivalQA: A Large-scale Benchmark Dataset for Open Domain Question Answering over Historical News Collections](https://mlsanity.com/paper/2109.03438/) (2021)
- [Let's Synthesize Step by Step: Iterative Dataset Synthesis with Large Language Models by Extrapolating Errors from Small Models](https://mlsanity.com/paper/2310.13671/) (2023)
- [Generative Language Models for Paragraph-Level Question Generation](https://mlsanity.com/paper/2210.03992/) (2022)
- [FewshotQA: A simple framework for few-shot learning of question answering tasks using pre-trained text-to-text models](https://mlsanity.com/paper/2109.01951/) (2021)
- [Improving Unsupervised Question Answering via Summarization-Informed Question Generation](https://mlsanity.com/paper/2109.07954/) (2021)
- [Contrastive Domain Adaptation for Question Answering using Limited Text Corpora](https://mlsanity.com/paper/2108.13854/) (2021)
- [Question Answering Infused Pre-training of General-Purpose Contextualized Representations](https://mlsanity.com/paper/2106.08190/) (2021)
- [LIQUID: A Framework for List Question Answering Dataset Generation](https://mlsanity.com/paper/2302.01691/) (2023)
- [SynthBio: A Case Study in Human-AI Collaborative Curation of Text Datasets](https://mlsanity.com/paper/2111.06467/) (2021)
- [Generating Information-Seeking Conversations from Unlabeled Documents](https://mlsanity.com/paper/2205.12609/) (2022)
- [RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Question](https://mlsanity.com/paper/2211.01482/) (2022)
- [Learning to Ask Like a Physician](https://mlsanity.com/paper/2206.02696/) (2022)
- [An Empirical Comparison of LM-based Question and Answer Generation Methods](https://mlsanity.com/paper/2305.17002/) (2023)
- [A synthetic data approach for domain generalization of NLI models](https://mlsanity.com/paper/2402.12368/) (2024)
- [UniGen: A Unified Generative Framework for Retrieval and Question Answering with Large Language Models](https://mlsanity.com/paper/2312.11036/) (2023)
- [textitTrans-LoRA: towards data-free Transferable Parameter Efficient Finetuning](https://mlsanity.com/paper/2405.17258/) (2024)

## Abstract (arXiv)

Question and answer generation is a data augmentation method that aims to improve question answering (QA) models given the limited amount of human labeled data. However, a considerable gap remains between synthetic and human-generated question-answer pairs. This work aims to narrow this gap by taking advantage of large language models and explores several factors such as model size, quality of pretrained models, scale of data synthesized, and algorithmic choices. On the SQuAD1.1 question answering task, we achieve higher accuracy using solely synthetic questions and answers than when using the SQuAD1.1 training set questions alone. Removing access to real Wikipedia data, we synthesize questions and answers from a synthetic corpus generated by an 8.3 billion parameter GPT-2 model. With no access to human supervision and only access to other models, we are able to train state of the art question answering networks on entirely model-generated data that achieve 88.4 Exact Match (EM) and 93.9 F1 score on the SQuAD1.1 dev set. We further apply our methodology to SQuAD2.0 and show a 2.8 absolute gain on EM score compared to prior work using synthetic data.