# Learning to Answer by Learning to Ask: Getting the Best of GPT-2 and BERT Worlds

arXiv: https://arxiv.org/abs/1911.02365 · HTML: https://mlsanity.com/paper/1911.02365/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Nov 2019.

## Built on by (most cited first, 19 of 19)

- [AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining](https://mlsanity.com/paper/2308.05734/) (2023)
- [Distilling Reasoning Capabilities into Smaller Language Models](https://mlsanity.com/paper/2212.00193/) (2022)
- [Automatic Generation of Socratic Subquestions for Teaching Math Word Problems](https://mlsanity.com/paper/2211.12835/) (2022)
- [Guiding Generative Language Models for Data Augmentation in Few-Shot Text Classification](https://mlsanity.com/paper/2111.09064/) (2021)
- [Large Human Language Models: A Need and the Challenges](https://mlsanity.com/paper/2312.07751/) (2023)
- [Multi-level Fusion of Wav2vec 2.0 and BERT for Multimodal Emotion Recognition](https://mlsanity.com/paper/2207.04697/) (2022)
- [Back-Training excels Self-Training at Unsupervised Domain Adaptation of Question Generation and Passage Retrieval](https://mlsanity.com/paper/2104.08801/) (2021)
- [Cooperative Self-training of Machine Reading Comprehension](https://mlsanity.com/paper/2103.07449/) (2021)
- [Combining Data Generation and Active Learning for Low-Resource Question Answering](https://mlsanity.com/paper/2211.14880/) (2022)
- [To Adapt or to Annotate: Challenges and Interventions for Domain Adaptation in Open-Domain Question Answering](https://mlsanity.com/paper/2212.10381/) (2022)
- [Automatic Generation and Evaluation of Reading Comprehension Test Items with Large Language Models](https://mlsanity.com/paper/2404.07720/) (2024)
- [Few-shot Question Generation for Personalized Feedback in Intelligent Tutoring Systems](https://mlsanity.com/paper/2206.04187/) (2022)
- [IDK-MRC: Unanswerable Questions for Indonesian Machine Reading Comprehension](https://mlsanity.com/paper/2210.13778/) (2022)
- [Question Answering and Question Generation for Finnish](https://mlsanity.com/paper/2211.13794/) (2022)
- [How to Build Robust FAQ Chatbot with Controllable Question Generator?](https://mlsanity.com/paper/2112.03007/) (2021)
- [Word Play for Playing Othello (Reverses)](https://mlsanity.com/paper/2207.08766/) (2022)
- [GenQ: Automated Question Generation to Support Caregivers While Reading Stories with Children](https://mlsanity.com/paper/2305.16809/) (2023)
- [Deceiving Question-Answering Models: A Hybrid Word-Level Adversarial Approach](https://mlsanity.com/paper/2411.08248/) (2024)
- [Subjective Question Generation and Answer Evaluation using NLP](https://mlsanity.com/paper/2512.17289/) (2025)

## Abstract (arXiv)

Automatic question generation aims at the generation of questions from a context, with the corresponding answers being sub-spans of the given passage. Whereas, most of the methods mostly rely on heuristic rules to generate questions, more recently also neural network approaches have been proposed. In this work, we propose a variant of the self-attention Transformer network architectures model to generate meaningful and diverse questions. To this end, we propose an easy to use model consisting of the conjunction of the Transformer decoder GPT-2 model with Transformer encoder BERT for the downstream task for question answering. The model is trained in an end-to-end fashion, where the language model is trained to produce a question-answer-aware input representation that facilitates to generate an answer focused question. Our result of neural question generation from text on the SQuAD 1.1 dataset suggests that our method can produce semantically correct and diverse questions. Additionally, we assessed the performance of our proposed method for the downstream task of question answering. The analysis shows that our proposed generation & answering collaboration framework relatively improves both tasks and is particularly powerful in the semi-supervised setup. The results further suggest a robust and comparably lean pipeline facilitating question generation in the small-data regime.