# End-to-End Video Question-Answer Generation with Generator-Pretester Network

arXiv: https://arxiv.org/abs/2101.01447 · HTML: https://mlsanity.com/paper/2101.01447/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 70 references that are in the map.
- First posted on arXiv 5 Jan 2021.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Unified Language Model Pre-training for Natural Language Understanding and Generation](https://mlsanity.com/paper/1905.03197/) (2019)
- [ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering](https://mlsanity.com/paper/1906.02467/) (2019)
- [CLEVRER: CoLlision Events for Video REpresentation and Reasoning](https://mlsanity.com/paper/1910.01442/) (2019)
- [TVQA: Localized, Compositional Video Question Answering](https://mlsanity.com/paper/1809.01696/) (2018)
- [TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering](https://mlsanity.com/paper/1704.04497/) (2017)
- [NewsQA: A Machine Comprehension Dataset](https://mlsanity.com/paper/1611.09830/) (2016)
- [OpenNMT: Open-source Toolkit for Neural Machine Translation](https://mlsanity.com/paper/1709.03815/) (2017)
- [End-to-End Dense Video Captioning with Masked Transformer](https://mlsanity.com/paper/1804.00819/) (2018)
- [Sequence to Sequence -- Video to Text](https://mlsanity.com/paper/1505.00487/) (2015)
- [Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering](https://mlsanity.com/paper/1904.04357/) (2019)
- [Motion-Appearance Co-Memory Networks for Video Question Answering](https://mlsanity.com/paper/1803.10906/) (2018)
- [Describing Videos by Exploiting Temporal Structure](https://mlsanity.com/paper/1502.08029/) (2015)
- [Leveraging Video Descriptions to Learn Video Question Answering](https://mlsanity.com/paper/1611.04021/) (2016)
- [Jointly Modeling Embedding and Translation to Bridge Video and Language](https://mlsanity.com/paper/1505.01861/) (2015)
- [Semantic Compositional Networks for Visual Captioning](https://mlsanity.com/paper/1611.08002/) (2016)
- [Video Captioning with Transferred Semantic Attributes](https://mlsanity.com/paper/1611.07675/) (2016)
- [Visual Question Generation as Dual Task of Visual Question Answering](https://mlsanity.com/paper/1709.07192/) (2017)
- [Information Maximizing Visual Question Generation](https://mlsanity.com/paper/1903.11207/) (2019)
- [Incorporating Global Visual Features into Attention-Based Neural Machine Translation](https://mlsanity.com/paper/1701.06521/) (2017)
- [Move Forward and Tell: A Progressive Generator of Video Descriptions](https://mlsanity.com/paper/1807.10018/) (2018)
- [iVQA: Inverse Visual Question Answering](https://mlsanity.com/paper/1710.03370/) (2017)

## Built on by (most cited first, 6 of 6)

- [Label Semantic Knowledge Distillation for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2208.03763/) (2022)
- [Language Models are Causal Knowledge Extractors for Zero-shot Video Question Answering](https://mlsanity.com/paper/2304.03754/) (2023)
- [CurEvo: Curriculum-Guided Self-Evolution for Video Understanding](https://mlsanity.com/paper/2604.26707/) (2026)
- [ECIS-VQG: Generation of Entity-centric Information-seeking Questions from Videos](https://mlsanity.com/paper/2410.09776/) (2024)
- [From text to multimodal: a survey of adversarial example generation in question answering systems](https://mlsanity.com/paper/2312.16156/) (2023)
- [ALIVE: An Avatar-Lecture Interactive Video Engine with Content-Aware Retrieval for Real-Time Interaction](https://mlsanity.com/paper/2512.20858/) (2025)

## Abstract (arXiv)

We study a novel task, Video Question-Answer Generation (VQAG), for challenging Video Question Answering (Video QA) task in multimedia. Due to expensive data annotation costs, many widely used, large-scale Video QA datasets such as Video-QA, MSVD-QA and MSRVTT-QA are automatically annotated using Caption Question Generation (CapQG) which inputs captions instead of the video itself. As captions neither fully represent a video, nor are they always practically available, it is crucial to generate question-answer pairs based on a video via Video Question-Answer Generation (VQAG). Existing video-to-text (V2T) approaches, despite taking a video as the input, only generate a question alone. In this work, we propose a novel model Generator-Pretester Network that focuses on two components: (1) The Joint Question-Answer Generator (JQAG) which generates a question with its corresponding answer to allow Video Question "Answering" training. (2) The Pretester (PT) verifies a generated question by trying to answer it and checks the pretested answer with both the model's proposed answer and the ground truth answer. We evaluate our system with the only two available large-scale human-annotated Video QA datasets and achieves state-of-the-art question generation performances. Furthermore, using our generated QA pairs only on the Video QA task, we can surpass some supervised baselines. We apply our generated questions to Video QA applications and surpasses some supervised baselines using generated questions only. As a pre-training strategy, we outperform both CapQG and transfer learning approaches when employing semi-supervised (20%) or fully supervised learning with annotated data. These experimental results suggest the novel perspectives for Video QA training.