# Fool Me Twice: Entailment from Wikipedia Gamification arXiv: https://arxiv.org/abs/2104.04725 ยท HTML: https://mlsanity.com/paper/2104.04725/ ## Key facts - Cited by 50 papers in the map (data through 14 Oct 2026). - Builds on 23 of its 44 references that are in the map. - First posted on arXiv 10 Apr 2021. ## Builds on (most cited first, 23 of 23) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019) - [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020) - [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017) - [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020) - [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015) - [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019) - [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017) - [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018) - [KILT: a Benchmark for Knowledge Intensive Language Tasks](https://mlsanity.com/paper/2009.02252/) (2020) - [Hypothesis Only Baselines in Natural Language Inference](https://mlsanity.com/paper/1805.01042/) (2018) - [Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases](https://mlsanity.com/paper/1909.03683/) (2019) - [HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verification](https://mlsanity.com/paper/2011.03088/) (2020) - [Towards Debiasing Fact Verification Models](https://mlsanity.com/paper/1908.05267/) (2019) - [Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension](https://mlsanity.com/paper/2002.00293/) (2020) - [Towards Debiasing NLU Models from Unknown Biases](https://mlsanity.com/paper/2009.12303/) (2020) - [Fine-grained Fact Verification with Kernel Graph Attention Network](https://mlsanity.com/paper/1910.09796/) (2019) - [GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification](https://mlsanity.com/paper/1908.01843/) (2019) - [Understanding tables with intermediate pre-training](https://mlsanity.com/paper/2010.00571/) (2020) - [Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailment](https://mlsanity.com/paper/1804.08117/) (2018) - [What Question Answering can Learn from Trivia Nerds](https://mlsanity.com/paper/1910.14464/) (2019) - [Asking Crowdworkers to Write Entailment Examples: The Best of Bad Options](https://mlsanity.com/paper/2010.06122/) (2020) - [Quizz: Targeted crowdsourcing with a billion (potential) users](https://mlsanity.com/paper/1506.01062/) (2015) ## Built on by (most cited first, 30 of 50) - [A Survey on Automated Fact-Checking](https://mlsanity.com/paper/2108.11896/) (2021) - [Generate rather than Retrieve: Large Language Models are Strong Context Generators](https://mlsanity.com/paper/2209.10063/) (2022) - [Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity](https://mlsanity.com/paper/2310.07521/) (2023) - [CommonsenseQA 2.0: Exposing the Limits of AI through Gamification](https://mlsanity.com/paper/2201.05320/) (2022) - [CREAK: A Dataset for Commonsense Reasoning over Entity Knowledge](https://mlsanity.com/paper/2109.01653/) (2021) - [A Survey of LLM-based Deep Search Agents: Paradigm, Optimization, Evaluation, and Challenges](https://mlsanity.com/paper/2508.05668/) (2025) - [Do Large Language Models Know about Facts?](https://mlsanity.com/paper/2310.05177/) (2023) - [Large Language Models Help Humans Verify Truthfulness -- Except When They Are Convincingly Wrong](https://mlsanity.com/paper/2310.12558/) (2023) - [Missing Counter-Evidence Renders NLP Fact-Checking Unrealistic for Misinformation](https://mlsanity.com/paper/2210.13865/) (2022) - [Synthetic Disinformation Attacks on Automated Fact Verification Systems](https://mlsanity.com/paper/2202.09381/) (2022) - [Mention Memory: incorporating textual knowledge into Transformers through entity mention attention](https://mlsanity.com/paper/2110.06176/) (2021) - [When Large Language Models Meet Vector Databases: A Survey](https://mlsanity.com/paper/2402.01763/) (2024) - [Augmented Large Language Models with Parametric Knowledge Guiding](https://mlsanity.com/paper/2305.04757/) (2023) - [Language Models Hallucinate, but May Excel at Fact Verification](https://mlsanity.com/paper/2310.14564/) (2023) - [FaVIQ: FAct Verification from Information-seeking Questions](https://mlsanity.com/paper/2107.02153/) (2021) - [EX-FEVER: A Dataset for Multi-hop Explainable Fact Verification](https://mlsanity.com/paper/2310.09754/) (2023) - [Trends in Integration of Knowledge and Large Language Models: A Survey and Taxonomy of Methods, Benchmarks, and Applications](https://mlsanity.com/paper/2311.05876/) (2023) - [Varifocal Question Generation for Fact-checking](https://mlsanity.com/paper/2210.12400/) (2022) - [Analyzing Dynamic Adversarial Training Data in the Limit](https://mlsanity.com/paper/2110.08514/) (2021) - [Claim Verification in the Age of Large Language Models: A Survey](https://mlsanity.com/paper/2408.14317/) (2024) - [CrossAug: A Contrastive Data Augmentation Method for Debiasing Fact Verification Models](https://mlsanity.com/paper/2109.15107/) (2021) - [MCTS-RAG: Enhancing Retrieval-Augmented Generation with Monte Carlo Tree Search](https://mlsanity.com/paper/2503.20757/) (2025) - [WinoGAViL: Gamified Association Benchmark to Challenge Vision-and-Language Models](https://mlsanity.com/paper/2207.12576/) (2022) - [Generative Large Language Models in Automated Fact-Checking: A Survey](https://mlsanity.com/paper/2407.02351/) (2024) - [Explainable Fact-checking through Question Answering](https://mlsanity.com/paper/2110.05369/) (2021) - [Hallucination to Truth: A Review of Fact-Checking and Factuality Evaluation in Large Language Models](https://mlsanity.com/paper/2508.03860/) (2025) - [Continually Improving Extractive QA via Human Feedback](https://mlsanity.com/paper/2305.12473/) (2023) - [EXPLAIN, EDIT, GENERATE: Rationale-Sensitive Counterfactual Data Augmentation for Multi-hop Fact Verification](https://mlsanity.com/paper/2310.14508/) (2023) - [More Victories, Less Cooperation: Assessing Cicero's Diplomacy Play](https://mlsanity.com/paper/2406.04643/) (2024) - [On Evaluating Explanation Utility for Human-AI Decision Making in NLP](https://mlsanity.com/paper/2407.03545/) (2024) ## Abstract (arXiv) We release FoolMeTwice (FM2 for short), a large dataset of challenging entailment pairs collected through a fun multi-player game. Gamification encourages adversarial examples, drastically lowering the number of examples that can be solved using "shortcuts" compared to other popular entailment datasets. Players are presented with two tasks. The first task asks the player to write a plausible claim based on the evidence from a Wikipedia page. The second one shows two plausible claims written by other players, one of which is false, and the goal is to identify it before the time runs out. Players "pay" to see clues retrieved from the evidence pool: the more evidence the player needs, the harder the claim. Game-play between motivated players leads to diverse strategies for crafting claims, such as temporal inference and diverting to unrelated evidence, and results in higher quality data for the entailment and evidence retrieval tasks. We open source the dataset and the game code.