# Say What You Mean! Large Language Models Speak Too Positively about Negative Commonsense Knowledge

arXiv: https://arxiv.org/abs/2305.05976 · HTML: https://mlsanity.com/paper/2305.05976/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 84 references that are in the map.
- First posted on arXiv 10 May 2023.

## Builds on (most cited first, 30 of 47)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [Learning To Retrieve Prompts for In-Context Learning](https://mlsanity.com/paper/2112.08633/) (2021)
- [Neural Text Generation with Unlikelihood Training](https://mlsanity.com/paper/1908.04319/) (2019)
- [e-SNLI: Natural Language Inference with Natural Language Explanations](https://mlsanity.com/paper/1812.01193/) (2018)
- [MetaICL: Learning to Learn In Context](https://mlsanity.com/paper/2110.15943/) (2021)
- [ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language](https://mlsanity.com/paper/2012.13048/) (2020)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [Transformers as Soft Reasoners over Language](https://mlsanity.com/paper/2002.05867/) (2020)
- [Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly](https://mlsanity.com/paper/1911.03343/) (2019)
- [How Context Affects Language Models' Factual Predictions](https://mlsanity.com/paper/2005.04611/) (2020)
- [Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations](https://mlsanity.com/paper/2205.11822/) (2022)
- [A Survey of Knowledge-Enhanced Text Generation](https://mlsanity.com/paper/2010.04389/) (2020)

## Built on by (most cited first, 30 of 32)

- [Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts](https://mlsanity.com/paper/2305.13300/) (2023)
- [Knowledge Conflicts for LLMs: A Survey](https://mlsanity.com/paper/2403.08319/) (2024)
- [Large Language Models are Temporal and Causal Reasoners for Video Question Answering](https://mlsanity.com/paper/2310.15747/) (2023)
- [Benchmarking Knowledge Boundary for Large Language Models: A Different Perspective on Model Evaluation](https://mlsanity.com/paper/2402.11493/) (2024)
- [UHGEval: Benchmarking the Hallucination of Chinese Large Language Models via Unconstrained Generation](https://mlsanity.com/paper/2311.15296/) (2023)
- [A Survey of Optimization-based Task and Motion Planning: From Classical To Learning Approaches](https://mlsanity.com/paper/2404.02817/) (2024)
- [This is not a Dataset: A Large Negation Benchmark to Challenge Large Language Models](https://mlsanity.com/paper/2310.15941/) (2023)
- [In Search of the Long-Tail: Systematic Generation of Long-Tail Inferential Knowledge via Logical Rule Guided Search](https://mlsanity.com/paper/2311.07237/) (2023)
- [AI, Meet Human: Learning Paradigms for Hybrid Decision Making Systems](https://mlsanity.com/paper/2402.06287/) (2024)
- [Leveraging Prompt-Based Large Language Models: Predicting Pandemic Health Decisions and Outcomes Through Social Media Language](https://mlsanity.com/paper/2403.00994/) (2024)
- [Every Answer Matters: Evaluating Commonsense with Probabilistic Measures](https://mlsanity.com/paper/2406.04145/) (2024)
- [Statistical Knowledge Assessment for Large Language Models](https://mlsanity.com/paper/2305.10519/) (2023)
- [How does Misinformation Affect Large Language Model Behaviors and Preferences?](https://mlsanity.com/paper/2505.21608/) (2025)
- [Thunder-NUBench: A Benchmark for LLMs' Sentence-Level Negation Understanding](https://mlsanity.com/paper/2506.14397/) (2025)
- [Generative AI for Self-Adaptive Systems: State of the Art and Research Roadmap](https://mlsanity.com/paper/2512.04680/) (2025)
- [KGQuiz: Evaluating the Generalization of Encoded Knowledge in Large Language Models](https://mlsanity.com/paper/2310.09725/) (2023)
- [Gold: A Global and Local-aware Denoising Framework for Commonsense Knowledge Graph Noise Detection](https://mlsanity.com/paper/2310.12011/) (2023)
- [Can large language models generate salient negative statements?](https://mlsanity.com/paper/2305.16755/) (2023)
- [Are LLMs Models of Distributional Semantics? A Case Study on Quantifiers](https://mlsanity.com/paper/2410.13984/) (2024)
- [SAGA: A Participant-specific Examination of Story Alternatives and Goal Applicability for a Deeper Understanding of Complex Events](https://mlsanity.com/paper/2408.05793/) (2024)
- [Snowman: A Million-scale Chinese Commonsense Knowledge Graph Distilled from Foundation Model](https://mlsanity.com/paper/2306.10241/) (2023)
- [BOOST: Harnessing Black-Box Control to Boost Commonsense in LMs' Generation](https://mlsanity.com/paper/2310.17054/) (2023)
- [Rule or Story, Which is a Better Commonsense Expression for Talking with Large Language Models?](https://mlsanity.com/paper/2402.14355/) (2024)
- [ACCORD: Closing the Commonsense Measurability Gap](https://mlsanity.com/paper/2406.02804/) (2024)
- [Talking the Talk Does Not Entail Walking the Walk: On the Limits of Large Language Models in Lexical Entailment Recognition](https://mlsanity.com/paper/2406.14894/) (2024)
- [Probing Causality Manipulation of Large Language Models](https://mlsanity.com/paper/2408.14380/) (2024)
- [Estimating Commonsense Plausibility through Semantic Shifts](https://mlsanity.com/paper/2502.13464/) (2025)
- [Common Sense vs. Morality: The Curious Case of Narrative Focus Bias in LLMs](https://mlsanity.com/paper/2603.09434/) (2026)
- [Carrot and Stick: Inducing Self-Motivation with Positive & Negative Feedback](https://mlsanity.com/paper/2406.16521/) (2024)
- [CoVeR: Conformal Calibration for Versatile and Reliable Autoregressive Next-Token Prediction](https://mlsanity.com/paper/2509.04733/) (2025)

## Abstract (arXiv)

Large language models (LLMs) have been widely studied for their ability to store and utilize positive knowledge. However, negative knowledge, such as "lions don't live in the ocean", is also ubiquitous in the world but rarely mentioned explicitly in the text. What do LLMs know about negative knowledge? This work examines the ability of LLMs to negative commonsense knowledge. We design a constrained keywords-to-sentence generation task (CG) and a Boolean question-answering task (QA) to probe LLMs. Our experiments reveal that LLMs frequently fail to generate valid sentences grounded in negative commonsense knowledge, yet they can correctly answer polar yes-or-no questions. We term this phenomenon the belief conflict of LLMs. Our further analysis shows that statistical shortcuts and negation reporting bias from language modeling pre-training cause this conflict.