Making Language Models Robust Against Negation
Not on the map. Why?
Abstract (arXiv)
Negation has been a long-standing challenge for language models. Previous studies have shown that they struggle with negation in many natural language understanding tasks. In this work, we propose a self-supervised method to make language models more robust against negation. We introduce a novel task, Next Sentence Polarity Prediction (NSPP), and a variation of the Next Sentence Prediction (NSP) task. We show that BERT and RoBERTa further pre-trained on our tasks outperform the off-the-shelf versions on nine negation-related benchmarks. Most notably, our pre-training tasks yield between 1.8% and 9.1% improvement on CondaQA, a large question-answering corpus requiring reasoning over negation.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 27 of its 59 references that are in the map. First posted on arXiv 11 Feb 2025.
Builds on
27 of 59 references.
References · 27
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
- 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
- 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
- 2019Language Models as Knowledge Bases?cited by 1,936
15 more
- 2015A large annotated corpus for learning natural language inferencecited by 1,847
- 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
- 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 372
- 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 320
- 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 243
- 2023Language models are not naysayers: An analysis of language models on negation benchmarkscited by 74
- 2021Understanding by Understanding Not: Modeling Negation in Language Modelscited by 73
- 2022UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Trainingcited by 57
- 2022An Analysis of Negation in Natural Language Understanding Corporacited by 39
- 2022CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negationcited by 39
- 2018Do Language Models Understand Anything? On the Ability of LSTMs to Understand Negative Polarity Itemscited by 36
- 2019NegBERT: A Transfer Learning Approach for Negation Detection and Scope Resolutioncited by 36
- 2022Improving negation detection with negation-focused pre-trainingcited by 14
- 2022Beyond Distributional Hypothesis: Let Language Models Learn Meaning-Text Correspondencecited by 10
- 2024Paraphrasing in Affirmative Terms Improves Negation Understandingcited by 8
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.