# COM2SENSE: A Commonsense Reasoning Benchmark with Complementary Sentences

arXiv: https://arxiv.org/abs/2106.00969 · HTML: https://mlsanity.com/paper/2106.00969/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 52 references that are in the map.
- First posted on arXiv 2 Jun 2021.

## Builds on (most cited first, 30 of 37)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [UnifiedQA: Crossing Format Boundaries With a Single QA System](https://mlsanity.com/paper/2005.00700/) (2020)
- [Learning the Difference that Makes a Difference with Counterfactually-Augmented Data](https://mlsanity.com/paper/1909.12434/) (2019)
- [A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories](https://mlsanity.com/paper/1604.01696/) (2016)
- [SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference](https://mlsanity.com/paper/1808.05326/) (2018)
- [Abductive Commonsense Reasoning](https://mlsanity.com/paper/1908.05739/) (2019)
- [Hypothesis Only Baselines in Natural Language Inference](https://mlsanity.com/paper/1805.01042/) (2018)
- [Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning](https://mlsanity.com/paper/1909.00277/) (2019)
- [ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension](https://mlsanity.com/paper/1810.12885/) (2018)
- [Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets](https://mlsanity.com/paper/1908.07898/) (2019)
- [Adversarial Filters of Dataset Biases](https://mlsanity.com/paper/2002.04108/) (2020)
- [Breaking NLI Systems with Sentences that Require Simple Lexical Inferences](https://mlsanity.com/paper/1805.02266/) (2018)
- [Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension](https://mlsanity.com/paper/2002.00293/) (2020)
- [Visual Question Answering: A Survey of Methods and Datasets](https://mlsanity.com/paper/1607.05910/) (2016)
- ["Going on a vacation" takes longer than "Going for a walk": A Study of Temporal Commonsense Understanding](https://mlsanity.com/paper/1909.03065/) (2019)
- [Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Models](https://mlsanity.com/paper/2005.00683/) (2020)
- [How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarks](https://mlsanity.com/paper/1808.04926/) (2018)

## Built on by (most cited first, 30 of 40)

- [Language is All a Graph Needs](https://mlsanity.com/paper/2308.07134/) (2023)
- [Challenges and Strategies in Cross-Cultural NLP](https://mlsanity.com/paper/2203.10020/) (2022)
- [Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations](https://mlsanity.com/paper/2205.11822/) (2022)
- [CLadder: Assessing Causal Reasoning in Language Models](https://mlsanity.com/paper/2312.04350/) (2023)
- [A Survey of Prompt Engineering Methods in Large Language Models for Different NLP Tasks](https://mlsanity.com/paper/2407.12994/) (2024)
- [Vera: A General-Purpose Plausibility Estimation Model for Commonsense Statements](https://mlsanity.com/paper/2305.03695/) (2023)
- [Analyzing Dataset Annotation Quality Management in the Wild](https://mlsanity.com/paper/2307.08153/) (2023)
- [Human Parity on CommonsenseQA: Augmenting Self-Attention with External Attention](https://mlsanity.com/paper/2112.03254/) (2021)
- [Benchmarks for Automated Commonsense Reasoning: A Survey](https://mlsanity.com/paper/2302.04752/) (2023)
- [CRASS: A Novel Data Set and Benchmark to Test Counterfactual Reasoning of Large Language Models](https://mlsanity.com/paper/2112.11941/) (2021)
- [Multi-Task Inference: Can Large Language Models Follow Multiple Instructions at Once?](https://mlsanity.com/paper/2402.11597/) (2024)
- [Cause and Effect: Can Large Language Models Truly Understand Causality?](https://mlsanity.com/paper/2402.18139/) (2024)
- [BIRD: A Trustworthy Bayesian Inference Framework for Large Language Models](https://mlsanity.com/paper/2404.12494/) (2024)
- [Reasoning on a Spectrum: Aligning LLMs to System 1 and System 2 Thinking](https://mlsanity.com/paper/2502.12470/) (2025)
- ["John is 50 years old, can his son be 65?" Evaluating NLP Models' Understanding of Feasibility](https://mlsanity.com/paper/2210.07471/) (2022)
- [Crystal: Introspective Reasoners Reinforced with Self-Feedback](https://mlsanity.com/paper/2310.04921/) (2023)
- [A Critical Review of Causal Reasoning Benchmarks for Large Language Models](https://mlsanity.com/paper/2407.08029/) (2024)
- [An Overview Of Temporal Commonsense Reasoning and Acquisition](https://mlsanity.com/paper/2308.00002/) (2023)
- [Always Tell Me The Odds: Fine-grained Conditional Probability Estimation](https://mlsanity.com/paper/2505.01595/) (2025)
- [CRoW: Benchmarking Commonsense Reasoning in Real-World Tasks](https://mlsanity.com/paper/2310.15239/) (2023)
- [ACQUIRED: A Dataset for Answering Counterfactual Questions In Real-Life Videos](https://mlsanity.com/paper/2311.01620/) (2023)
- [Commonsense Reasoning for Conversational AI: A Survey of the State of the Art](https://mlsanity.com/paper/2302.07926/) (2023)
- [STRUX: An LLM for Decision-Making with Structured Explanations](https://mlsanity.com/paper/2410.12583/) (2024)
- [CARE-MI: Chinese Benchmark for Misinformation Evaluation in Maternity and Infant Care](https://mlsanity.com/paper/2307.01458/) (2023)
- [CryptoX : Compositional Reasoning Evaluation of Large Language Models](https://mlsanity.com/paper/2502.07813/) (2025)
- [VIPHY: Probing "Visible" Physical Commonsense Knowledge](https://mlsanity.com/paper/2209.07000/) (2022)
- [Towards CausalGPT: A Multi-Agent Approach for Faithful Knowledge Reasoning via Promoting Causal Consistency in LLMs](https://mlsanity.com/paper/2308.11914/) (2023)
- [Can NLP Models 'Identify', 'Distinguish', and 'Justify' Questions that Don't have a Definitive Answer?](https://mlsanity.com/paper/2309.04635/) (2023)
- [SituatedGen: Incorporating Geographical and Temporal Contexts into Generative Commonsense Reasoning](https://mlsanity.com/paper/2306.12552/) (2023)
- [TEXT2AFFORD: Probing Object Affordance Prediction abilities of Language Models solely from Text](https://mlsanity.com/paper/2402.12881/) (2024)

## Abstract (arXiv)

Commonsense reasoning is intuitive for humans but has been a long-term challenge for artificial intelligence (AI). Recent advancements in pretrained language models have shown promising results on several commonsense benchmark datasets. However, the reliability and comprehensiveness of these benchmarks towards assessing model's commonsense reasoning ability remains unclear. To this end, we introduce a new commonsense reasoning benchmark dataset comprising natural language true/false statements, with each sample paired with its complementary counterpart, resulting in 4k sentence pairs. We propose a pairwise accuracy metric to reliably measure an agent's ability to perform commonsense reasoning over a given situation. The dataset is crowdsourced and enhanced with an adversarial model-in-the-loop setup to incentivize challenging samples. To facilitate a systematic analysis of commonsense capabilities, we design our dataset along the dimensions of knowledge domains, reasoning scenarios and numeracy. Experimental results demonstrate that our strongest baseline (UnifiedQA-3B), after fine-tuning, achieves ~71% standard accuracy and ~51% pairwise accuracy, well below human performance (~95% for both metrics). The dataset is available at https://github.com/PlusLabNLP/Com2Sense.