# This is not correct! Negation-aware Evaluation of Language Generation Systems

arXiv: https://arxiv.org/abs/2307.13989 · HTML: https://mlsanity.com/paper/2307.13989/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 28 references that are in the map.
- First posted on arXiv 26 Jul 2023.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [MTEB: Massive Text Embedding Benchmark](https://mlsanity.com/paper/2210.07316/) (2022)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly](https://mlsanity.com/paper/1911.03343/) (2019)
- [Learning Compact Metrics for MT](https://mlsanity.com/paper/2110.06341/) (2021)
- [Understanding by Understanding Not: Modeling Negation in Language Models](https://mlsanity.com/paper/2105.03519/) (2021)
- [Perturbation CheckLists for Evaluating NLG Evaluation Metrics](https://mlsanity.com/paper/2109.05771/) (2021)
- [NegBERT: A Transfer Learning Approach for Negation Detection and Scope Resolution](https://mlsanity.com/paper/1911.04211/) (2019)
- [DEMETR: Diagnosing Evaluation Metrics for Translation](https://mlsanity.com/paper/2210.13746/) (2022)
- [Not another Negation Benchmark: The NaN-NLI Test Suite for Sub-clausal Negation](https://mlsanity.com/paper/2210.03256/) (2022)
- [Improving negation detection with negation-focused pre-training](https://mlsanity.com/paper/2205.04012/) (2022)
- [Multi-task Learning of Negation and Speculation for Targeted Sentiment Classification](https://mlsanity.com/paper/2010.08318/) (2020)

## Built on by (most cited first, 17 of 17)

- [Exploring the Impact of Temperature on Large Language Models:Hot or Cold?](https://mlsanity.com/paper/2506.07295/) (2025)
- [Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practices](https://mlsanity.com/paper/2408.09169/) (2024)
- [How and where does CLIP process negation?](https://mlsanity.com/paper/2407.10488/) (2024)
- [Negation: A Pink Elephant in the Large Language Models' Room?](https://mlsanity.com/paper/2503.22395/) (2025)
- [Quantifying Cognitive Bias Induction in LLM-Generated Content](https://mlsanity.com/paper/2507.03194/) (2025)
- [Thunder-NUBench: A Benchmark for LLMs' Sentence-Level Negation Understanding](https://mlsanity.com/paper/2506.14397/) (2025)
- [CASA: Causality-driven Argument Sufficiency Assessment](https://mlsanity.com/paper/2401.05249/) (2024)
- [PARC: A Quantitative Framework Uncovering the Symmetries within Vision Language Models](https://mlsanity.com/paper/2506.14808/) (2025)
- [CliniBench: A Clinical Outcome Prediction Benchmark for Generative and Encoder-Based Language Models](https://mlsanity.com/paper/2509.26136/) (2025)
- [Semantic Adapter for Universal Text Embeddings: Diagnosing and Mitigating Negation Blindness to Enhance Universality](https://mlsanity.com/paper/2504.00584/) (2025)
- [Explainable Semantic Textual Similarity via Dissimilar Span Detection](https://mlsanity.com/paper/2603.21174/) (2026)
- [Similarity Gates Approve Reversals: A Validity Audit of Embedding-Cosine Thresholds in Agent Systems](https://mlsanity.com/paper/2608.10216/) (2026)
- [Real-World Summarization: When Evaluation Reaches Its Limits](https://mlsanity.com/paper/2507.11508/) (2025)
- [NegBLEURT Forest: Leveraging Inconsistencies for Detecting Jailbreak Attacks](https://mlsanity.com/paper/2511.11784/) (2025)
- [Examining the Metrics for Document-Level Claim Extraction in Czech and Slovak](https://mlsanity.com/paper/2511.14566/) (2025)
- [The Effect of Negation on CLIP in Medical Imaging: Limitations of Contrastive Language-Image Pretraining](https://mlsanity.com/paper/2512.17121/) (2025)
- [Isotropic Yet Undecodable: The Sequential Content-Sufficiency Gap in Latent-Predictive Text Representations](https://mlsanity.com/paper/2610.07906/) (2026)

## Abstract (arXiv)

Large language models underestimate the impact of negations on how much they change the meaning of a sentence. Therefore, learned evaluation metrics based on these models are insensitive to negations. In this paper, we propose NegBLEURT, a negation-aware version of the BLEURT evaluation metric. For that, we designed a rule-based sentence negation tool and used it to create the CANNOT negation evaluation dataset. Based on this dataset, we fine-tuned a sentence transformer and an evaluation metric to improve their negation sensitivity. Evaluating these models on existing benchmarks shows that our fine-tuned models outperform existing metrics on the negated sentences by far while preserving their base models' performances on other perturbations.