mlsanityData to 7 Oct 2026

CommonsenseQA 2.0: Exposing the Limits of AI through Gamification

arXiv 2201.05320 ↗First posted 14 Jan 2022cs.CL cs.AI cs.LGcited by 108

Not on the map. Why?

Cited by 48 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Constructing benchmarks that test the abilities of modern natural language understanding models is difficult - pre-trained language models exploit artifacts in benchmarks to achieve human parity, but still fail on adversarial examples and make errors that demonstrate a lack of common sense. In this work, we propose gamification as a framework for data construction. The goal of players in the game is to compose questions that mislead a rival AI while using specific phrases for extra points. The game environment leads to enhanced user engagement and simultaneously gives the game designer control over the collected data, allowing us to collect high-quality data at scale. Using our method we create CommonsenseQA 2.0, which includes 14,343 yes/no questions, and demonstrate its difficulty for models that are orders-of-magnitude larger than the AI used in the game itself. Our best baseline, the T5-based Unicorn with 11B parameters achieves an accuracy of 70.2%, substantially higher than GPT-3 (52.9%) in a few-shot inference setup. Both score well below human performance which is at 94.1%.

Key facts. Cited by 108 papers in the map (data through 7 Oct 2026). Builds on 40 of its 64 references that are in the map. First posted on arXiv 14 Jan 2022.

Builds on

40 of 64 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  8. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  9. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  10. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  11. 2015Teaching Machines to Read and Comprehendcited by 1,039
  12. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
28 more
  1. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  2. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  3. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  4. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  5. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 428
  6. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 423
  7. 2018The Web as a Knowledge-base for Answering Complex Questionscited by 409
  8. 2017Adversarial Examples: Attacks and Defenses for Deep Learningcited by 350
  9. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  10. 2018Adversarially Robust Generalization Requires More Datacited by 318
  11. 2019WinoGrande: An Adversarial Winograd Schema Challenge at Scalecited by 194
  12. 2019MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehensioncited by 183
  13. 2021MultiModalQA: Complex Question Answering over Text, Tables and Imagescited by 177
  14. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
  15. 2020Adversarial Filters of Dataset Biasescited by 159
  16. 2020An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Modelscited by 133
  17. 2020Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehensioncited by 131
  18. 2019Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoningcited by 100
  19. 2021UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmarkcited by 98
  20. 2019Compositional Questions Do Not Necessitate Multi-hop Reasoningcited by 87
  21. 2019Reasoning Over Paragraph Effects in Situationscited by 79
  22. 2016A Thorough Examination of the CNN/Daily Mail Reading Comprehension Taskcited by 71
  23. 2019MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehensioncited by 60
  24. 2021Fool Me Twice: Entailment from Wikipedia Gamificationcited by 50
  25. 2017Towards Linguistically Generalizable NLP Systems: A Workshop and Shared Taskcited by 24
  26. 2020Leap-Of-Thought: Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledgecited by 24
  27. 2017Mastering the Dungeon: Grounded Language Learning by Mechanical Turker Descentcited by 10
  28. 2015Quizz: Targeted crowdsourcing with a billion (potential) userscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 21; 2023: 34; 2024: 28; 2025: 21; 2026: 4.

108 papers.

Sort:

By area:

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
  3. 2024Scaling and evaluating sparse autoencoderscited by 530
  4. 2022UL2: Unifying Language Learning Paradigmscited by 292
  5. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 200
  6. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  7. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  8. 2022Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanationscited by 159
  9. 2023Causal Parrots: Large Language Models May Talk Causality But Are Not Causalcited by 146
  10. 2025Are Sparse Autoencoders Useful? A Case Study in Sparse Probingcited by 109
  11. 2024LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Modelscited by 97
  12. 2023Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debatecited by 92
38 more
  1. 2022Mind's Eye: Grounded Language Model Reasoning through Simulationcited by 83
  2. 2023How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questionscited by 82
  3. 2023Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Imagescited by 79
  4. 2023Natural Language Reasoning, A Surveycited by 61
  5. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 51
  6. 2024A Survey of Prompt Engineering Methods in Large Language Models for Different NLP Taskscited by 46
  7. 2023Vera: A General-Purpose Plausibility Estimation Model for Commonsense Statementscited by 44
  8. 2023Benchmarks for Automated Commonsense Reasoning: A Surveycited by 40
  9. 2023Are Machine Rationales (Not) Useful to Humans? Measuring and Improving Human Utility of Free-Text Rationalescited by 37
  10. 2025A Rosetta Stone for AI Benchmarkscited by 29
  11. 2023Trends in Integration of Knowledge and Large Language Models: A Survey and Taxonomy of Methods, Benchmarks, and Applicationscited by 28
  12. 2024Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and Improving LLMscited by 28
  13. 2024CulturalTeaming: AI-Assisted Interactive Red-Teaming for Challenging LLMs' (Lack of) Multicultural Knowledgecited by 28
  14. 2023Second Thoughts are Best: Learning to Re-Align With Human Values from Text Editscited by 27
  15. 2023CAT: A Contextualized Conceptualization and Instantiation Framework for Commonsense Reasoningcited by 25
  16. 2024K-Level Reasoning: Establishing Higher Order Beliefs in Large Language Models for Strategic Reasoningcited by 21
  17. 2022WinoGAViL: Gamified Association Benchmark to Challenge Vision-and-Language Modelscited by 20
  18. 2023Chain-of-Skills: A Configurable Model for Open-domain Question Answeringcited by 19
  19. 2022Unpacking Large Language Models with Conceptual Consistencycited by 18
  20. 2024MALMM: Multi-Agent Large Language Models for Zero-Shot Robotics Manipulationcited by 18
  21. 2025Uncertainty Quantification for Language Models: A Suite of Black-Box, White-Box, LLM Judge, and Ensemble Scorerscited by 16
  22. 2022Task Compass: Scaling Multi-task Pre-training with Task Prefixcited by 14
  23. 2022Open-domain Question Answering via Chain of Reasoning over Heterogeneous Knowledgecited by 14
  24. 2022Retrieval Augmentation for Commonsense Reasoning: A Unified Approachcited by 14
  25. 2024CRAFT Your Dataset: Task-Specific Synthetic Dataset Generation Through Corpus Retrieval and Augmentationcited by 14
  26. 2025LLMs Can't Handle Peer Pressure: Crumbling under Multi-Agent Social Interactionscited by 14
  27. 2024LLMScan: Causal Scan for LLM Misbehavior Detectioncited by 13
  28. 2024A Survey on Human-Centric LLMscited by 13
  29. 2023PathFinder: Guided Search over Multi-Step Reasoning Pathscited by 12
  30. 2023LaMPP: Language Models as Probabilistic Priors for Perception and Actioncited by 11
  31. 2023Generalizable Chain-of-Thought Prompting in Mixed-task Scenarios with Large Language Modelscited by 11
  32. 2022Two is Better than Many? Binary Classification as an Effective Approach to Multi-Choice Question Answeringcited by 10
  33. 2024mCSQA: Multilingual Commonsense Reasoning Dataset with Unified Creation Strategy by Language Models and Humanscited by 10
  34. 2024Visual Riddles: a Commonsense and World Knowledge Challenge for Large Vision and Language Modelscited by 10
  35. 2022Inferring Implicit Relations in Complex Questions with Language Modelscited by 9
  36. 2022Great Truths are Always Simple: A Rather Simple Knowledge Encoder for Enhancing the Commonsense Reasoning Capacity of Pre-Trained Modelscited by 9
  37. 2022Elaboration-Generating Commonsense Question Answering at Scalecited by 9
  38. 2023Toward Grounded Commonsense Reasoningcited by 9

50 most cited of 108 All 108 (JSON)