# ProtoQA: A Question Answering Dataset for Prototypical Common-Sense Reasoning

arXiv: https://arxiv.org/abs/2005.00771 · HTML: https://mlsanity.com/paper/2005.00771/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 May 2020.

## Built on by (most cited first, 30 of 46)

- [MetaICL: Learning to Learn In Context](https://mlsanity.com/paper/2110.15943/) (2021)
- [Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models](https://mlsanity.com/paper/2203.06904/) (2022)
- [CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP](https://mlsanity.com/paper/2104.08835/) (2021)
- [ChatGPT is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Models](https://mlsanity.com/paper/2303.16421/) (2023)
- [RiddleSense: Reasoning about Riddle Questions Featuring Linguistic Creativity and Commonsense Knowledge](https://mlsanity.com/paper/2101.00376/) (2021)
- [Natural Language Reasoning, A Survey](https://mlsanity.com/paper/2303.14725/) (2023)
- [NormBank: A Knowledge Bank of Situational Social Norms](https://mlsanity.com/paper/2305.17008/) (2023)
- [ExplaGraphs: An Explanation Graph Generation Task for Structured Commonsense Reasoning](https://mlsanity.com/paper/2104.07644/) (2021)
- [A Systematic Investigation of Commonsense Knowledge in Large Language Models](https://mlsanity.com/paper/2111.00607/) (2021)
- [Benchmarks for Automated Commonsense Reasoning: A Survey](https://mlsanity.com/paper/2302.04752/) (2023)
- [Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Models](https://mlsanity.com/paper/2412.02980/) (2024)
- [INQUIRE: A Natural World Text-to-Image Retrieval Benchmark](https://mlsanity.com/paper/2411.02537/) (2024)
- [CulturalBench: A Robust, Diverse, and Challenging Cultural Benchmark by Human-AI CulturalTeaming](https://mlsanity.com/paper/2410.02677/) (2024)
- [Dimensions of Commonsense Knowledge](https://mlsanity.com/paper/2101.04640/) (2021)
- [MAQA: Evaluating Uncertainty Quantification in LLMs Regarding Data Uncertainty](https://mlsanity.com/paper/2408.06816/) (2024)
- [Exploring Universal Intrinsic Task Subspace via Prompt Tuning](https://mlsanity.com/paper/2110.07867/) (2021)
- [BRAINTEASER: Lateral Thinking Puzzles for Large Language Models](https://mlsanity.com/paper/2310.05057/) (2023)
- [Penguins Don't Fly: Reasoning about Generics through Instantiations and Exceptions](https://mlsanity.com/paper/2205.11658/) (2022)
- [LLMs Can't Handle Peer Pressure: Crumbling under Multi-Agent Social Interactions](https://mlsanity.com/paper/2508.18321/) (2025)
- [Extracting Biomedical Factual Knowledge Using Pretrained Language Model and Electronic Health Record Context](https://mlsanity.com/paper/2209.07859/) (2022)
- [Eliciting and Understanding Cross-Task Skills with Task-Level Mixture-of-Experts](https://mlsanity.com/paper/2205.12701/) (2022)
- [Exploring Strategies for Generalizable Commonsense Reasoning with Pre-trained Models](https://mlsanity.com/paper/2109.02837/) (2021)
- [Learning to Initialize: Can Meta Learning Improve Cross-task Generalization in Prompt Tuning?](https://mlsanity.com/paper/2302.08143/) (2023)
- [Right for Right Reasons: Large Language Models for Verifiable Commonsense Knowledge Graph Question Answering](https://mlsanity.com/paper/2403.01390/) (2024)
- [Always Tell Me The Odds: Fine-grained Conditional Probability Estimation](https://mlsanity.com/paper/2505.01595/) (2025)
- [Few-shot Adaptation Works with UnpredicTable Data](https://mlsanity.com/paper/2208.01009/) (2022)
- [Towards Concept-Aware Large Language Models](https://mlsanity.com/paper/2311.01866/) (2023)
- [On the Tip of the Tongue: Analyzing Conceptual Representation in Large Language Models with Reverse-Dictionary Probe](https://mlsanity.com/paper/2402.14404/) (2024)
- [Every Answer Matters: Evaluating Commonsense with Probabilistic Measures](https://mlsanity.com/paper/2406.04145/) (2024)
- [Beyond Output Matching: Bidirectional Alignment for Enhanced In-Context Learning](https://mlsanity.com/paper/2312.17055/) (2023)

## Abstract (arXiv)

Given questions regarding some prototypical situation such as Name something that people usually do before they leave the house for work? a human can easily answer them via acquired experiences. There can be multiple right answers for such questions, with some more common for a situation than others. This paper introduces a new question answering dataset for training and evaluating common sense reasoning capabilities of artificial intelligence systems in such prototypical situations. The training set is gathered from an existing set of questions played in a long-running international game show FAMILY- FEUD. The hidden evaluation set is created by gathering answers for each question from 100 crowd-workers. We also propose a generative evaluation task where a model has to output a ranked list of answers, ideally covering all prototypical answers for a question. After presenting multiple competitive baseline models, we find that human performance still exceeds model scores on all evaluation metrics with a meaningful gap, supporting the challenging nature of the task.