# What Gives the Answer Away? Question Answering Bias Analysis on Video QA Datasets

arXiv: https://arxiv.org/abs/2007.03626 · HTML: https://mlsanity.com/paper/2007.03626/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jul 2020.

## Built on by (most cited first, 13 of 13)

- [VideoAgent: Long-form Video Understanding with Large Language Model as Agent](https://mlsanity.com/paper/2403.10517/) (2024)
- [A Simple LLM Framework for Long-Range Video Question-Answering](https://mlsanity.com/paper/2312.17235/) (2023)
- [AGQA: A Benchmark for Compositional Spatio-Temporal Reasoning](https://mlsanity.com/paper/2103.16002/) (2021)
- [AMEGO: Active Memory from long EGOcentric videos](https://mlsanity.com/paper/2409.10917/) (2024)
- [PACS: A Dataset for Physical Audiovisual CommonSense Reasoning](https://mlsanity.com/paper/2203.11130/) (2022)
- [ANetQA: A Large-scale Benchmark for Fine-grained Compositional Reasoning over Untrimmed Videos](https://mlsanity.com/paper/2305.02519/) (2023)
- [Video Question Answering with Phrases via Semantic Roles](https://mlsanity.com/paper/2104.03762/) (2021)
- [AGQA 2.0: An Updated Benchmark for Compositional Spatio-Temporal Reasoning](https://mlsanity.com/paper/2204.06105/) (2022)
- [Mitigating Easy Option Bias in Multiple-Choice Question Answering](https://mlsanity.com/paper/2508.13428/) (2025)
- [TrUMAn: Trope Understanding in Movies and Animations](https://mlsanity.com/paper/2108.04542/) (2021)
- [NarrativeXL: A Large-scale Dataset For Long-Term Memory Models](https://mlsanity.com/paper/2305.13877/) (2023)
- [QTG-VQA: Question-Type-Guided Architectural for VideoQA Systems](https://mlsanity.com/paper/2409.09348/) (2024)
- [TempCloze: Can Video-LLMs Identify the Missing Middle?](https://mlsanity.com/paper/2609.01515/) (2026)

## Abstract (arXiv)

Question answering biases in video QA datasets can mislead multimodal model to overfit to QA artifacts and jeopardize the model's ability to generalize. Understanding how strong these QA biases are and where they come from helps the community measure progress more accurately and provide researchers insights to debug their models. In this paper, we analyze QA biases in popular video question answering datasets and discover pretrained language models can answer 37-48% questions correctly without using any multimodal context information, far exceeding the 20% random guess baseline for 5-choose-1 multiple-choice questions. Our ablation study shows biases can come from annotators and type of questions. Specifically, annotators that have been seen during training are better predicted by the model and reasoning, abstract questions incur more biases than factual, direct questions. We also show empirically that using annotator-non-overlapping train-test splits can reduce QA biases for video QA datasets.