# An Analysis of Negation in Natural Language Understanding Corpora

arXiv: https://arxiv.org/abs/2203.08929 · HTML: https://mlsanity.com/paper/2203.08929/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 37 references that are in the map.
- First posted on arXiv 16 Mar 2022.

## Builds on (most cited first, 19 of 19)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://mlsanity.com/paper/1908.07125/) (2019)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations](https://mlsanity.com/paper/1808.09121/) (2018)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [Stress Test Evaluation for Natural Language Inference](https://mlsanity.com/paper/1806.00692/) (2018)
- [Coreference Resolution as Query-based Span Prediction](https://mlsanity.com/paper/1911.01746/) (2019)
- [Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIs](https://mlsanity.com/paper/1909.02597/) (2019)
- [jiant: A Software Toolkit for Research on General-Purpose Text Understanding Models](https://mlsanity.com/paper/2003.02249/) (2020)
- [NegBERT: A Transfer Learning Approach for Negation Detection and Scope Resolution](https://mlsanity.com/paper/1911.04211/) (2019)
- [HELP: A Dataset for Identifying Shortcomings of Neural Models in Monotonicity Reasoning](https://mlsanity.com/paper/1904.12166/) (2019)

## Built on by (most cited first, 30 of 39)

- [Evaluating Large Language Models at Evaluating Instruction Following](https://mlsanity.com/paper/2310.07641/) (2023)
- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)
- [Language models are not naysayers: An analysis of language models on negation benchmarks](https://mlsanity.com/paper/2306.08189/) (2023)
- [CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation](https://mlsanity.com/paper/2211.00295/) (2022)
- [Say What You Mean! Large Language Models Speak Too Positively about Negative Commonsense Knowledge](https://mlsanity.com/paper/2305.05976/) (2023)
- [NevIR: Negation in Neural Information Retrieval](https://mlsanity.com/paper/2305.07614/) (2023)
- [It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performance](https://mlsanity.com/paper/2305.09022/) (2023)
- [Not another Negation Benchmark: The NaN-NLI Test Suite for Sub-clausal Negation](https://mlsanity.com/paper/2210.03256/) (2022)
- [On the Limitations of Dataset Balancing: The Lost Battle Against Spurious Correlations](https://mlsanity.com/paper/2204.12708/) (2022)
- [This is not a Dataset: A Large Negation Benchmark to Challenge Large Language Models](https://mlsanity.com/paper/2310.15941/) (2023)
- [Investigating and Addressing Hallucinations of LLMs in Tasks Involving Negation](https://mlsanity.com/paper/2406.05494/) (2024)
- [Beyond Positive Scaling: How Negation Impacts Scaling Trends of Language Models](https://mlsanity.com/paper/2305.17311/) (2023)
- [A negation detection assessment of GPTs: analysis with the xNot360 dataset](https://mlsanity.com/paper/2306.16638/) (2023)
- [Paraphrasing in Affirmative Terms Improves Negation Understanding](https://mlsanity.com/paper/2406.07492/) (2024)
- [Leveraging Affirmative Interpretations from Negation Improves Natural Language Understanding](https://mlsanity.com/paper/2210.14486/) (2022)
- [Simple Linguistic Inferences of Large Language Models (LLMs): Blind Spots and Blinds](https://mlsanity.com/paper/2305.14785/) (2023)
- [The Dark Side of Rich Rewards: Understanding and Mitigating Noise in VLM Rewards](https://mlsanity.com/paper/2409.15922/) (2024)
- [Multi-Agent LLM Judge: automatic personalized LLM judge design for evaluating natural language generation applications](https://mlsanity.com/paper/2504.02867/) (2025)
- [Limited Linguistic Diversity in Embodied AI Datasets](https://mlsanity.com/paper/2601.03136/) (2026)
- [Exploring Lottery Prompts for Pre-trained Language Models](https://mlsanity.com/paper/2305.19500/) (2023)
- [Thunder-NUBench: A Benchmark for LLMs' Sentence-Level Negation Understanding](https://mlsanity.com/paper/2506.14397/) (2025)
- [Do Methods to Jailbreak and Defend LLMs Generalize Across Languages?](https://mlsanity.com/paper/2511.00689/) (2025)
- [MAQA: A Multimodal QA Benchmark for Negation](https://mlsanity.com/paper/2301.03238/) (2023)
- [Balancing Exploration and Exploitation in LLM using Soft RLLF for Enhanced Negation Understanding](https://mlsanity.com/paper/2403.01185/) (2024)
- [Making Language Models Robust Against Negation](https://mlsanity.com/paper/2502.07717/) (2025)
- [Revisiting subword tokenization: A case study on affixal negation in large language models](https://mlsanity.com/paper/2404.02421/) (2024)
- [SAGA: A Participant-specific Examination of Story Alternatives and Goal Applicability for a Deeper Understanding of Complex Events](https://mlsanity.com/paper/2408.05793/) (2024)
- [Investigating Counterclaims in Causality Extraction from Text](https://mlsanity.com/paper/2510.08224/) (2025)
- [A Framework For Refining Text Classification and Object Recognition from Academic Articles](https://mlsanity.com/paper/2305.17401/) (2023)
- [Snowman: A Million-scale Chinese Commonsense Knowledge Graph Distilled from Foundation Model](https://mlsanity.com/paper/2306.10241/) (2023)

## Abstract (arXiv)

This paper analyzes negation in eight popular corpora spanning six natural language understanding tasks. We show that these corpora have few negations compared to general-purpose English, and that the few negations in them are often unimportant. Indeed, one can often ignore negations and still make the right predictions. Additionally, experimental results show that state-of-the-art transformers trained with these corpora obtain substantially worse results with instances that contain negation, especially if the negations are important. We conclude that new corpora accounting for negation are needed to solve natural language understanding tasks when negation is present.