# What Question Answering can Learn from Trivia Nerds

arXiv: https://arxiv.org/abs/1910.14464 · HTML: https://mlsanity.com/paper/1910.14464/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Oct 2019.

## Built on by (most cited first, 21 of 21)

- [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://mlsanity.com/paper/2308.16884/) (2023)
- [What Will it Take to Fix Benchmarking in Natural Language Understanding?](https://mlsanity.com/paper/2104.02145/) (2021)
- [QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension](https://mlsanity.com/paper/2107.12708/) (2021)
- [FiD-Light: Efficient and Effective Retrieval-Augmented Text Generation](https://mlsanity.com/paper/2209.14290/) (2022)
- [NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned](https://mlsanity.com/paper/2101.00133/) (2021)
- [Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Above](https://mlsanity.com/paper/2502.14127/) (2025)
- [Fool Me Twice: Entailment from Wikipedia Gamification](https://mlsanity.com/paper/2104.04725/) (2021)
- [What's in a Name? Answer Equivalence For Open-Domain Question Answering](https://mlsanity.com/paper/2109.05289/) (2021)
- [PaniniQA: Enhancing Patient Education Through Interactive Question Answering](https://mlsanity.com/paper/2308.03253/) (2023)
- [Is That Your Final Answer? Test-Time Scaling Improves Selective Question Answering](https://mlsanity.com/paper/2502.13962/) (2025)
- [Accurate and Nuanced Open-QA Evaluation Through Textual Entailment](https://mlsanity.com/paper/2405.16702/) (2024)
- [A Survey on Deep Learning Event Extraction: Approaches and Applications](https://mlsanity.com/paper/2107.02126/) (2021)
- [RAAT: Relation-Augmented Attention Transformer for Relation Modeling in Document-Level Event Extraction](https://mlsanity.com/paper/2206.03377/) (2022)
- [Capturing Humans' Mental Models of AI: An Item Response Theory Approach](https://mlsanity.com/paper/2305.09064/) (2023)
- [Toward Deconfounding the Influence of Entity Demographics for Question Answering Accuracy](https://mlsanity.com/paper/2104.07571/) (2021)
- [A Russian Jeopardy! Data Set for Question-Answering Systems](https://mlsanity.com/paper/2112.02325/) (2021)
- [Fighting FIRe with FIRE: Assessing the Validity of Text-to-Video Retrieval Benchmarks](https://mlsanity.com/paper/2210.05038/) (2022)
- [On Degrees of Freedom in Defining and Testing Natural Language Understanding](https://mlsanity.com/paper/2305.15130/) (2023)
- [How the Advent of Ubiquitous Large Language Models both Stymie and Turbocharge Dynamic Adversarial Question Generation](https://mlsanity.com/paper/2401.11185/) (2024)
- [AUDITA: A New Dataset to Audit Humans vs. AI Skill at Audio QA](https://mlsanity.com/paper/2604.21766/) (2026)
- [CFMatch: Aligning Automated Answer Equivalence Evaluation with Expert Judgments For Open-Domain Question Answering](https://mlsanity.com/paper/2401.13170/) (2024)

## Abstract (arXiv)

In addition to the traditional task of getting machines to answer questions, a major research question in question answering is to create interesting, challenging questions that can help systems learn how to answer questions and also reveal which systems are the best at answering questions. We argue that creating a question answering dataset -- and the ubiquitous leaderboard that goes with it -- closely resembles running a trivia tournament: you write questions, have agents (either humans or machines) answer the questions, and declare a winner. However, the research community has ignored the decades of hard-learned lessons from decades of the trivia community creating vibrant, fair, and effective question answering competitions. After detailing problems with existing QA datasets, we outline the key lessons -- removing ambiguity, discriminating skill, and adjudicating disputes -- that can transfer to QA research and how they might be implemented for the QA community.