mlsanityData to 7 Oct 2026

Language models are not naysayers: An analysis of language models on negation benchmarks

arXiv 2306.08189 ↗First posted 14 Jun 2023cs.CLcited by 73

Not on the map. Why?

Abstract (arXiv)

Negation has been shown to be a major bottleneck for masked language models, such as BERT. However, whether this finding still holds for larger-sized auto-regressive language models (``LLMs'') has not been studied comprehensively. With the ever-increasing volume of research and applications of LLMs, we take a step back to evaluate the ability of current-generation LLMs to handle negation, a fundamental linguistic phenomenon that is central to language understanding. We evaluate different LLMs -- including the open-source GPT-neo, GPT-3, and InstructGPT -- against a wide range of negation benchmarks. Through systematic experimentation with varying model sizes and prompts, we show that LLMs have several limitations including insensitivity to the presence of negation, an inability to capture the lexical semantics of negation, and a failure to reason under negation.

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Builds on 39 of its 49 references that are in the map. First posted on arXiv 14 Jun 2023.

Builds on

39 of 49 references.

References · 39

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  10. 2022Emergent Abilities of Large Language Modelscited by 2,483
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  12. 2019Language Models as Knowledge Bases?cited by 1,922
27 more
  1. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  2. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  3. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  4. 2019How Can We Know What Language Models Know?cited by 953
  5. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
  6. 2020The Pushshift Reddit Datasetcited by 289
  7. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  8. 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 240
  9. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 215
  10. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 199
  11. 2020On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behaviorcited by 115
  12. 2022Inverse scaling can become U-shapedcited by 76
  13. 2022PromptBERT: Improving BERT Sentence Embeddings with Promptscited by 75
  14. 2021Understanding by Understanding Not: Modeling Negation in Language Modelscited by 73
  15. 2019Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIscited by 58
  16. 2022Can Large Language Models Truly Understand Prompts? A Case Study with Negated Promptscited by 56
  17. 2022SNCSE: Contrastive Learning for Unsupervised Sentence Embedding with Soft Negative Samplescited by 39
  18. 2022An Analysis of Negation in Natural Language Understanding Corporacited by 39
  19. 2022CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negationcited by 38
  20. 2023Consistency Analysis of ChatGPTcited by 37
  21. 2021Language Models Use Monotonicity to Assess NPI Licensingcited by 30
  22. 2022Not another Negation Benchmark: The NaN-NLI Test Suite for Sub-clausal Negationcited by 24
  23. 2022Improving negation detection with negation-focused pre-trainingcited by 14
  24. 2020Compositional and Lexical Semantics in RoBERTa, BERT and DistilBERT: A Case Study on CoQAcited by 12
  25. 2022Beyond Distributional Hypothesis: Let Language Models Learn Meaning-Text Correspondencecited by 10
  26. 2022Leveraging Affirmative Interpretations from Negation Improves Natural Language Understandingcited by 7
  27. 2022Life after BERT: What do Other Muppets Understand about Language?cited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 30; 2025: 22; 2026: 15.

73 papers.

Sort:

By area:

  1. 2024ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabiccited by 87
  2. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Surveycited by 82
  3. 2025Vision-Language Models Do Not Understand Negationcited by 69
  4. 2024Can LLMs Replace Manual Annotation of Software Engineering Artifacts?cited by 40
  5. 2024LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraintscited by 34
  6. 2023Are Multilingual LLMs Culturally-Diverse Reasoners? An Investigation into Multicultural Proverbs and Sayingscited by 32
  7. 2024Large Language Models Lack Understanding of Character Composition of Wordscited by 21
  8. 2024Learning to Ask Informative Questions: Enhancing LLMs with Preference Optimization and Expected Information Gaincited by 21
  9. 2025Large Language Models Do Not Simulate Human Psychologycited by 17
  10. 2024How and where does CLIP process negation?cited by 16
  11. 2024Language in Vivo vs. in Silico: Size Matters but Larger Language Models Still Do Not Comprehend Language on a Par with Humans Due to Impenetrable Semantic Referencecited by 15
  12. 2024Investigating and Addressing Hallucinations of LLMs in Tasks Involving Negationcited by 15
38 more
  1. 2024Enhancing Post-Hoc Attributions in Long Document Comprehension via Coarse Grained Answer Decompositioncited by 12
  2. 2023Can Language Models Be Tricked by Language Illusions? Easier with Syntax, Harder with Semanticscited by 11
  3. 2024SyntaxShap: Syntax-aware Explainability Method for Text Generationcited by 11
  4. 2024Is This a Bad Table? A Closer Look at the Evaluation of Table Generation from Textcited by 8
  5. 2024Paraphrasing in Affirmative Terms Improves Negation Understandingcited by 7
  6. 2024The Dark Side of Rich Rewards: Understanding and Mitigating Noise in VLM Rewardscited by 7
  7. 2025Negation: A Pink Elephant in the Large Language Models' Room?cited by 7
  8. 2023Revealing the Unwritten: Visual Investigation of Beam Search Trees to Address Language Model Prompting Challengescited by 6
  9. 2023Assessing Step-by-Step Reasoning against Lexical Negation: A Case Study on Syllogismcited by 6
  10. 2024Strong hallucinations from negation and how to fix themcited by 6
  11. 2025Don't Deceive Me: Mitigating Gaslighting through Attention Reallocation in LMMscited by 6
  12. 2025TNG-CLIP:Training-Time Negation Data Generation for Negation Awareness of CLIPcited by 6
  13. 2025Benchmarking Gaslighting Negation Attacks Against Reasoning Modelscited by 6
  14. 2025SinhalaMMLU: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in Sinhalacited by 6
  15. 2026Physics-IQ Verifiedcited by 6
  16. 2024Experimental Pragmatics with Machines: Testing LLM Predictions for the Inferences of Plain and Embedded Disjunctionscited by 5
  17. 2025Thunder-NUBench: A Benchmark for LLMs' Sentence-Level Negation Understandingcited by 5
  18. 2025FESTA: Functionally Equivalent Sampling for Trust Assessment of Multimodal LLMscited by 5
  19. 2024Large Language Models are Skeptics: False Negative Problem of Input-conflicting Hallucinationcited by 4
  20. 2024On A Scale From 1 to 5: Quantifying Hallucination in Faithfulness Evaluationcited by 4
  21. 2024Is artificial intelligence still intelligence? LLMs generalize to novel adjective-noun pairs, but don't mimic the full human distributioncited by 4
  22. 2025Prompting from the bench: Large-scale pretraining is not sufficient to prepare LLMs for ordinary meaning analysiscited by 4
  23. 2024NeIn: Telling What You Don't Wantcited by 3
  24. 2024Controlled LLM-based Reasoning for Clinical Trial Retrievalcited by 3
  25. 2024Are LLMs Models of Distributional Semantics? A Case Study on Quantifierscited by 3
  26. 2025Benchmarking Gaslighting Negation Attacks Against Multimodal Large Language Modelscited by 3
  27. 2025Making Language Models Robust Against Negationcited by 3
  28. 2024Exploring Group and Symmetry Principles in Large Language Modelscited by 2
  29. 2024Revisiting subword tokenization: A case study on affixal negation in large language modelscited by 2
  30. 2024Modelling Language using Large Language Modelscited by 2
  31. 2024Challenges and Opportunities in Text Generation Explainabilitycited by 2
  32. 2025Learning "Partner-Aware" Collaborators in Multi-Party Collaborationcited by 2
  33. 2025Syntactic Framing Fragility: An Audit of Robustness in LLM Ethical Decisionscited by 2
  34. 2025From No to Know: Taxonomy, Challenges, and Opportunities for Negation Understanding in Multimodal Foundation Modelscited by 1
  35. 2025Propositional Logic for Probing Generalization in Neural Networkscited by 1
  36. 2025Identifying a Circuit for Verb Conjugation in GPT-2cited by 1
  37. 2025QueryBandits for Hallucination Mitigation: Exploiting Semantic Features for No-Regret Rewritingcited by 1
  38. 2025Music Recommendation with Large Language Models: Challenges, Opportunities, and Evaluationcited by 1

50 most cited of 73 All 73 (JSON)