# Improving negation detection with negation-focused pre-training

arXiv: https://arxiv.org/abs/2205.04012 · HTML: https://mlsanity.com/paper/2205.04012/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 28 references that are in the map.
- First posted on arXiv 9 May 2022.

## Builds on (most cited first, 10 of 10)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](https://mlsanity.com/paper/2010.01057/) (2020)
- [NegBio: a high-performance tool for negation and uncertainty detection in radiology reports](https://mlsanity.com/paper/1712.05898/) (2017)
- [NegBERT: A Transfer Learning Approach for Negation Detection and Scope Resolution](https://mlsanity.com/paper/1911.04211/) (2019)
- [Improving Sentiment Analysis with Multi-task Learning of Negation](https://mlsanity.com/paper/1906.07610/) (2019)

## Built on by (most cited first, 14 of 14)

- [Language models are not naysayers: An analysis of language models on negation benchmarks](https://mlsanity.com/paper/2306.08189/) (2023)
- [Large Language Models Only Pass Primary School Exams in Indonesia: A Comprehensive Test on IndoMMLU](https://mlsanity.com/paper/2310.04928/) (2023)
- [CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation](https://mlsanity.com/paper/2211.00295/) (2022)
- [Not another Negation Benchmark: The NaN-NLI Test Suite for Sub-clausal Negation](https://mlsanity.com/paper/2210.03256/) (2022)
- [This is not correct! Negation-aware Evaluation of Language Generation Systems](https://mlsanity.com/paper/2307.13989/) (2023)
- [Beyond Positive Scaling: How Negation Impacts Scaling Trends of Language Models](https://mlsanity.com/paper/2305.17311/) (2023)
- [Paraphrasing in Affirmative Terms Improves Negation Understanding](https://mlsanity.com/paper/2406.07492/) (2024)
- [Simple Linguistic Inferences of Large Language Models (LLMs): Blind Spots and Blinds](https://mlsanity.com/paper/2305.14785/) (2023)
- [Are LLMs Models of Distributional Semantics? A Case Study on Quantifiers](https://mlsanity.com/paper/2410.13984/) (2024)
- [Making Language Models Robust Against Negation](https://mlsanity.com/paper/2502.07717/) (2025)
- [Semantic Adapter for Universal Text Embeddings: Diagnosing and Mitigating Negation Blindness to Enhance Universality](https://mlsanity.com/paper/2504.00584/) (2025)
- [Learning to Detect Cross-Modal Negation: An Analysis of Latent Representations and an Attention-Based Solution](https://mlsanity.com/paper/2607.17712/) (2026)
- [Negation Triplet Extraction with Syntactic Dependency and Semantic Consistency](https://mlsanity.com/paper/2404.09830/) (2024)
- [Not all Negation Cues are Equal: Affixal Negations Yield Better Negation Understanding](https://mlsanity.com/paper/2609.13685/) (2026)

## Abstract (arXiv)

Negation is a common linguistic feature that is crucial in many language understanding tasks, yet it remains a hard problem due to diversity in its expression in different types of text. Recent work has shown that state-of-the-art NLP models underperform on samples containing negation in various tasks, and that negation detection models do not transfer well across domains. We propose a new negation-focused pre-training strategy, involving targeted data augmentation and negation masking, to better incorporate negation information into language models. Extensive experiments on common benchmarks show that our proposed approach improves negation detection performance and generalizability over the strong baseline NegBERT (Khandewal and Sawant, 2020).