# CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation

arXiv: https://arxiv.org/abs/2211.00295 · HTML: https://mlsanity.com/paper/2211.00295/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 93 references that are in the map.
- First posted on arXiv 1 Nov 2022.

## Builds on (most cited first, 30 of 49)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://mlsanity.com/paper/2111.02080/) (2021)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://mlsanity.com/paper/1704.04683/) (2017)
- [UnifiedQA: Crossing Format Boundaries With a Single QA System](https://mlsanity.com/paper/2005.00700/) (2020)
- [Learning the Difference that Makes a Difference with Counterfactually-Augmented Data](https://mlsanity.com/paper/1909.12434/) (2019)
- [True Few-Shot Learning with Language Models](https://mlsanity.com/paper/2105.11447/) (2021)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly](https://mlsanity.com/paper/1911.03343/) (2019)
- [Stress Test Evaluation for Natural Language Inference](https://mlsanity.com/paper/1806.00692/) (2018)
- [The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning](https://mlsanity.com/paper/2205.03401/) (2022)
- [Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models](https://mlsanity.com/paper/2106.13353/) (2021)
- [Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning](https://mlsanity.com/paper/1908.05803/) (2019)

## Built on by (most cited first, 30 of 39)

- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [Language models are not naysayers: An analysis of language models on negation benchmarks](https://mlsanity.com/paper/2306.08189/) (2023)
- [Natural Language Reasoning, A Survey](https://mlsanity.com/paper/2303.14725/) (2023)
- [Say What You Mean! Large Language Models Speak Too Positively about Negative Commonsense Knowledge](https://mlsanity.com/paper/2305.05976/) (2023)
- [NevIR: Negation in Neural Information Retrieval](https://mlsanity.com/paper/2305.07614/) (2023)
- [Characterizing LLM Abstention Behavior in Science QA with Context Perturbations](https://mlsanity.com/paper/2404.12452/) (2024)
- [Learning to Ask Informative Questions: Enhancing LLMs with Preference Optimization and Expected Information Gain](https://mlsanity.com/paper/2406.17453/) (2024)
- [ScoNe: Benchmarking Negation Reasoning in Language Models With Fine-Tuning and In-Context Learning](https://mlsanity.com/paper/2305.19426/) (2023)
- [Prompt-Reverse Inconsistency: LLM Self-Inconsistency Beyond Generative Randomness and Prompt Paraphrasing](https://mlsanity.com/paper/2504.01282/) (2025)
- [Reverse Question Answering: Can an LLM Write a Question so Hard (or Bad) that it Can't Answer?](https://mlsanity.com/paper/2410.15512/) (2024)
- [Beyond Positive Scaling: How Negation Impacts Scaling Trends of Language Models](https://mlsanity.com/paper/2305.17311/) (2023)
- [Multi-dimensional Evaluation of Empathetic Dialog Responses](https://mlsanity.com/paper/2402.11409/) (2024)
- [Natural Language Processing RELIES on Linguistics](https://mlsanity.com/paper/2405.05966/) (2024)
- [WinoPron: Revisiting English Winogender Schemas for Consistency, Coverage, and Grammatical Case](https://mlsanity.com/paper/2409.05653/) (2024)
- [What Has Been Lost with Synthetic Evaluation?](https://mlsanity.com/paper/2505.22830/) (2025)
- [Paraphrasing in Affirmative Terms Improves Negation Understanding](https://mlsanity.com/paper/2406.07492/) (2024)
- [Evaluating Paraphrastic Robustness in Textual Entailment Models](https://mlsanity.com/paper/2306.16722/) (2023)
- [Whispers of Doubt Amidst Echoes of Triumph in NLP Robustness](https://mlsanity.com/paper/2311.09694/) (2023)
- [Thunder-NUBench: A Benchmark for LLMs' Sentence-Level Negation Understanding](https://mlsanity.com/paper/2506.14397/) (2025)
- [Can You Follow Me? Testing Situational Understanding in ChatGPT](https://mlsanity.com/paper/2310.16135/) (2023)
- [A Comprehensive Taxonomy of Negation for NLP and Neural Retrievers](https://mlsanity.com/paper/2507.22337/) (2025)
- [Balancing Exploration and Exploitation in LLM using Soft RLLF for Enhanced Negation Understanding](https://mlsanity.com/paper/2403.01185/) (2024)
- [NeIn: Telling What You Don't Want](https://mlsanity.com/paper/2409.06481/) (2024)
- [Making Language Models Robust Against Negation](https://mlsanity.com/paper/2502.07717/) (2025)
- [FLUKE: A Linguistically-Driven and Task-Agnostic Framework for Robustness Evaluation](https://mlsanity.com/paper/2504.17311/) (2025)
- [Revisiting subword tokenization: A case study on affixal negation in large language models](https://mlsanity.com/paper/2404.02421/) (2024)
- [PromptPrism: A Linguistically-Inspired Taxonomy for Prompts](https://mlsanity.com/paper/2505.12592/) (2025)
- [Investigating Counterclaims in Causality Extraction from Text](https://mlsanity.com/paper/2510.08224/) (2025)
- [How Much Consistency Is Your Accuracy Worth?](https://mlsanity.com/paper/2310.13781/) (2023)

## Abstract (arXiv)

The full power of human language-based communication cannot be realized without negation. All human languages have some form of negation. Despite this, negation remains a challenging phenomenon for current natural language understanding systems. To facilitate the future development of models that can process negation effectively, we present CONDAQA, the first English reading comprehension dataset which requires reasoning about the implications of negated statements in paragraphs. We collect paragraphs with diverse negation cues, then have crowdworkers ask questions about the implications of the negated statement in the passage. We also have workers make three kinds of edits to the passage -- paraphrasing the negated statement, changing the scope of the negation, and reversing the negation -- resulting in clusters of question-answer pairs that are difficult for models to answer with spurious shortcuts. CONDAQA features 14,182 question-answer pairs with over 200 unique negation cues and is challenging for current state-of-the-art models. The best performing model on CONDAQA (UnifiedQA-v2-3b) achieves only 42% on our consistency metric, well below human performance which is 81%. We release our dataset, along with fully-finetuned, few-shot, and zero-shot evaluations, to facilitate the development of future NLP methods that work on negated language.