mlsanityData to 7 Oct 2026

ProtoQA: A Question Answering Dataset for Prototypical Common-Sense Reasoning

arXiv 2005.00771 ↗First posted 2 May 2020cs.CLcited by 45

Not on the map. Why?

Abstract (arXiv)

Given questions regarding some prototypical situation such as Name something that people usually do before they leave the house for work? a human can easily answer them via acquired experiences. There can be multiple right answers for such questions, with some more common for a situation than others. This paper introduces a new question answering dataset for training and evaluating common sense reasoning capabilities of artificial intelligence systems in such prototypical situations. The training set is gathered from an existing set of questions played in a long-running international game show FAMILY- FEUD. The hidden evaluation set is created by gathering answers for each question from 100 crowd-workers. We also propose a generative evaluation task where a model has to output a ranked list of answers, ideally covering all prototypical answers for a question. After presenting multiple competitive baseline models, we find that human performance still exceeds model scores on all evaluation metrics with a meaningful gap, supporting the challenging nature of the task.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 9; 2023: 13; 2024: 8; 2025: 4; 2026: 2.

45 papers.

Sort:

By area:

  1. 2021MetaICL: Learning to Learn In Contextcited by 450
  2. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  3. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
  4. 2023ChatGPT is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Modelscited by 63
  5. 2023Natural Language Reasoning, A Surveycited by 61
  6. 2021RiddleSense: Reasoning about Riddle Questions Featuring Linguistic Creativity and Commonsense Knowledgecited by 60
  7. 2023NormBank: A Knowledge Bank of Situational Social Normscited by 60
  8. 2021ExplaGraphs: An Explanation Graph Generation Task for Structured Commonsense Reasoningcited by 58
  9. 2021A Systematic Investigation of Commonsense Knowledge in Large Language Modelscited by 56
  10. 2023Benchmarks for Automated Commonsense Reasoning: A Surveycited by 40
  11. 2024INQUIRE: A Natural World Text-to-Image Retrieval Benchmarkcited by 33
  12. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 31
33 more
  1. 2021Dimensions of Commonsense Knowledgecited by 29
  2. 2024CulturalBench: A Robust, Diverse, and Challenging Cultural Benchmark by Human-AI CulturalTeamingcited by 27
  3. 2021Exploring Universal Intrinsic Task Subspace via Prompt Tuningcited by 25
  4. 2024MAQA: Evaluating Uncertainty Quantification in LLMs Regarding Data Uncertaintycited by 25
  5. 2023BRAINTEASER: Lateral Thinking Puzzles for Large Language Modelscited by 18
  6. 2022Penguins Don't Fly: Reasoning about Generics through Instantiations and Exceptionscited by 17
  7. 2022Extracting Biomedical Factual Knowledge Using Pretrained Language Model and Electronic Health Record Contextcited by 15
  8. 2025LLMs Can't Handle Peer Pressure: Crumbling under Multi-Agent Social Interactionscited by 14
  9. 2022Eliciting and Understanding Cross-Task Skills with Task-Level Mixture-of-Expertscited by 12
  10. 2021Exploring Strategies for Generalizable Commonsense Reasoning with Pre-trained Modelscited by 11
  11. 2023Learning to Initialize: Can Meta Learning Improve Cross-task Generalization in Prompt Tuning?cited by 11
  12. 2024Right for Right Reasons: Large Language Models for Verifiable Commonsense Knowledge Graph Question Answeringcited by 11
  13. 2025Always Tell Me The Odds: Fine-grained Conditional Probability Estimationcited by 9
  14. 2022Few-shot Adaptation Works with UnpredicTable Datacited by 6
  15. 2023Towards Concept-Aware Large Language Modelscited by 6
  16. 2024On the Tip of the Tongue: Analyzing Conceptual Representation in Large Language Models with Reverse-Dictionary Probecited by 6
  17. 2024Every Answer Matters: Evaluating Commonsense with Probabilistic Measurescited by 6
  18. 2023Beyond Output Matching: Bidirectional Alignment for Enhanced In-Context Learningcited by 5
  19. 2025When Incentives Backfire, Data Stops Being Humancited by 4
  20. 2022Generalizable Neuro-symbolic Systems for Commonsense Question Answeringcited by 3
  21. 2022Leveraging pre-trained language models for conversational information seeking from textcited by 3
  22. 2023KEPR: Knowledge Enhancement and Plausibility Ranking for Generative Commonsense Question Answeringcited by 2
  23. 2023CommonsenseVIS: Visualizing and Understanding Commonsense Reasoning Capabilities of Natural Language Modelscited by 2
  24. 2023Enabling High-Level Machine Reasoning with Cognitive Neuro-Symbolic Systemscited by 2
  25. 2024ActionCOMET: A Zero-shot Approach to Learn Image-specific Commonsense Concepts about Actionscited by 2
  26. 2025SCoRE: Benchmarking Long-Chain Reasoning in Commonsense Scenarioscited by 2
  27. 2021Semantic Categorization of Social Knowledge for Commonsense Question Answeringcited by 1
  28. 2022DiscoSense: Commonsense Reasoning with Discourse Connectivescited by 1
  29. 2022Different Tunes Played with Equal Skill: Exploring a Unified Optimization Subspace for Delta Tuningcited by 1
  30. 2023Large Language Models Are Also Good Prototypical Commonsense Reasonerscited by 1
  31. 2023EXMODD: An EXplanatory Multimodal Open-Domain Dialogue datasetcited by 0
  32. 2026Human-Alignment, Calibration, and Activation Patterns in Large Language Model Uncertaintycited by 0
  33. 2026Improving the Diversity of LLM Outputs without a Trade-offcited by 0