mlsanityData to 14 Oct 2026

Thunder-NUBench: A Benchmark for LLMs' Sentence-Level Negation Understanding

arXiv 2506.14397 ↗First posted 17 Jun 2025cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Negation is a fundamental linguistic phenomenon that poses ongoing challenges for Large Language Models (LLMs), particularly in tasks requiring deep semantic understanding. Current benchmarks often treat negation as a minor detail within broader tasks, such as natural language inference. Consequently, there is a lack of benchmarks specifically designed to evaluate comprehension of negation. In this work, we introduce Thunder-NUBench, a novel benchmark explicitly created to assess sentence-level understanding of negation in LLMs. Thunder-NUBench goes beyond merely identifying surface-level cues by contrasting standard negation with structurally diverse alternatives, such as local negation, contradiction, and paraphrase. This benchmark includes manually curated sentence-negation pairs and a multiple-choice dataset, allowing for a comprehensive evaluation of models' understanding of negation.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 48 of its 116 references that are in the map. First posted on arXiv 17 Jun 2025.

Builds on

48 of 116 references.

References · 48

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  8. 2024Qwen2.5 Technical Reportcited by 4,942
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  11. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  12. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
36 more
  1. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  2. 2025Gemma 3 Technical Reportcited by 1,824
  3. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  4. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  5. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  6. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  7. 2015An Introduction to Convolutional Neural Networkscited by 514
  8. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 320
  9. 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 243
  10. 2018Stress Test Evaluation for Natural Language Inferencecited by 213
  11. 2020HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verificationcited by 201
  12. 2023Language models are not naysayers: An analysis of language models on negation benchmarkscited by 74
  13. 2021Understanding by Understanding Not: Modeling Negation in Language Modelscited by 73
  14. 2025Vision-Language Models Do Not Understand Negationcited by 72
  15. 2019Distributional Semantics and Linguistic Theorycited by 53
  16. 2023Better Zero-Shot Reasoning with Self-Adaptive Promptingcited by 52
  17. 2023A Practical Survey on Zero-shot Prompt Design for In-context Learningcited by 49
  18. 2019Can neural networks understand monotonicity reasoning?cited by 45
  19. 2022An Analysis of Negation in Natural Language Understanding Corporacited by 39
  20. 2022CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negationcited by 39
  21. 2018Do Language Models Understand Anything? On the Ability of LSTMs to Understand Negative Polarity Itemscited by 36
  22. 2019NegBERT: A Transfer Learning Approach for Negation Detection and Scope Resolutioncited by 36
  23. 2023Say What You Mean! Large Language Models Speak Too Positively about Negative Commonsense Knowledgecited by 32
  24. 2025Know "No" Better: A Data-Driven Approach for Enhancing Negation Awareness in CLIPcited by 28
  25. 2021"I'm Not Mad": Commonsense Implications of Negation and Contradictioncited by 26
  26. 2022Not another Negation Benchmark: The NaN-NLI Test Suite for Sub-clausal Negationcited by 25
  27. 2021A comparative evaluation and analysis of three generations of Distributional Semantic Modelscited by 19
  28. 2023ScoNe: Benchmarking Negation Reasoning in Language Models With Fine-Tuning and In-Context Learningcited by 18
  29. 2023This is not correct! Negation-aware Evaluation of Language Generation Systemscited by 17
  30. 2024How and where does CLIP process negation?cited by 16
  31. 2022Beyond Distributional Hypothesis: Let Language Models Learn Meaning-Text Correspondencecited by 10
  32. 2023A negation detection assessment of GPTs: analysis with the xNot360 datasetcited by 10
  33. 2024Paraphrasing in Affirmative Terms Improves Negation Understandingcited by 8
  34. 2022Leveraging Affirmative Interpretations from Negation Improves Natural Language Understandingcited by 7
  35. 2022A Question-Answer Driven Approach to Reveal Affirmative Interpretations from Verbal Negationscited by 5
  36. 2024The Self-Contained Negation Test Setcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025Don't Think of the White Bear: Ironic Negation in Transformer Models Under Cognitive Loadcited by 2
  2. 2026Learning to Detect Cross-Modal Negation: An Analysis of Latent Representations and an Attention-Based Solutioncited by 1
  3. 2026Thunder-KoNUBench: A Corpus-Aligned Benchmark for Korean Negation Understandingcited by 0
  4. 2026Framing Instability in LLM Ethical Stance: Auditing Negation Sensitivity in Moral Dilemmascited by 0
  5. 2026Benchmarking LLM Competence on Logical Inference over Probability Operatorscited by 0