# Can Large Language Models Truly Understand Prompts? A Case Study with Negated Prompts

arXiv: https://arxiv.org/abs/2209.12711 · HTML: https://mlsanity.com/paper/2209.12711/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 26 references that are in the map.
- First posted on arXiv 26 Sep 2022.

## Builds on (most cited first, 21 of 21)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://mlsanity.com/paper/1606.06031/) (2016)
- [A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories](https://mlsanity.com/paper/1604.01696/) (2016)
- [Do Prompt-Based Models Really Understand the Meaning of their Prompts?](https://mlsanity.com/paper/2109.01247/) (2021)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts](https://mlsanity.com/paper/2202.01279/) (2022)
- [Impact of Pretraining Term Frequencies on Few-Shot Reasoning](https://mlsanity.com/paper/2202.07206/) (2022)

## Built on by (most cited first, 30 of 59)

- [Evaluating Large Language Models at Evaluating Instruction Following](https://mlsanity.com/paper/2310.07641/) (2023)
- [Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters](https://mlsanity.com/paper/2212.10001/) (2022)
- [Programming Refusal with Conditional Activation Steering](https://mlsanity.com/paper/2409.05907/) (2024)
- [Personality Traits in Large Language Models](https://mlsanity.com/paper/2307.00184/) (2023)
- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)
- [Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Models](https://mlsanity.com/paper/2310.05338/) (2023)
- [Language models are not naysayers: An analysis of language models on negation benchmarks](https://mlsanity.com/paper/2306.08189/) (2023)
- [Enhancing Conversational Search: Large Language Model-Aided Informative Query Rewriting](https://mlsanity.com/paper/2310.09716/) (2023)
- [Active Learning Principles for In-Context Learning with Large Language Models](https://mlsanity.com/paper/2305.14264/) (2023)
- [Controlled Text Generation via Language Model Arithmetic](https://mlsanity.com/paper/2311.14479/) (2023)
- [Do Models Really Learn to Follow Instructions? An Empirical Study of Instruction Tuning](https://mlsanity.com/paper/2305.11383/) (2023)
- [You don't need a personality test to know these models are unreliable: Assessing the Reliability of Large Language Models on Psychometric Instruments](https://mlsanity.com/paper/2311.09718/) (2023)
- [Small Models are Valuable Plug-ins for Large Language Models](https://mlsanity.com/paper/2305.08848/) (2023)
- [Social Bias Evaluation for Large Language Models Requires Prompt Variations](https://mlsanity.com/paper/2407.03129/) (2024)
- [Say What You Mean! Large Language Models Speak Too Positively about Negative Commonsense Knowledge](https://mlsanity.com/paper/2305.05976/) (2023)
- [Large Language Model Instruction Following: A Survey of Progresses and Challenges](https://mlsanity.com/paper/2303.10475/) (2023)
- [Instruction-following Evaluation through Verbalizer Manipulation](https://mlsanity.com/paper/2307.10558/) (2023)
- [ALCM: Autonomous LLM-Augmented Causal Discovery Framework](https://mlsanity.com/paper/2405.01744/) (2024)
- [The Life Cycle of Knowledge in Big Language Models: A Survey](https://mlsanity.com/paper/2303.07616/) (2023)
- [Investigating and Addressing Hallucinations of LLMs in Tasks Involving Negation](https://mlsanity.com/paper/2406.05494/) (2024)
- [Are Language Models Worse than Humans at Following Prompts? It's Complicated](https://mlsanity.com/paper/2301.07085/) (2023)
- [Despite "super-human" performance, current LLMs are unsuited for decisions about ethics and safety](https://mlsanity.com/paper/2212.06295/) (2022)
- [Beyond Positive Scaling: How Negation Impacts Scaling Trends of Language Models](https://mlsanity.com/paper/2305.17311/) (2023)
- [Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Data](https://mlsanity.com/paper/2503.01167/) (2025)
- [CAT-LLM: Style-enhanced Large Language Models with Text Style Definition for Chinese Article-style Transfer](https://mlsanity.com/paper/2401.05707/) (2024)
- [Using Large Language Models to Automate and Expedite Reinforcement Learning with Reward Machine](https://mlsanity.com/paper/2402.07069/) (2024)
- [Generative AI in Knowledge Work: Design Implications for Data Navigation and Decision-Making](https://mlsanity.com/paper/2503.18419/) (2025)
- [Generalization Analogies: A Testbed for Generalizing AI Oversight to Hard-To-Measure Domains](https://mlsanity.com/paper/2311.07723/) (2023)
- [Say More with Less: Understanding Prompt Learning Behaviors through Gist Compression](https://mlsanity.com/paper/2402.16058/) (2024)
- [Emergent inabilities? Inverse scaling over the course of pretraining](https://mlsanity.com/paper/2305.14681/) (2023)

## Abstract (arXiv)

Previous work has shown that there exists a scaling law between the size of Language Models (LMs) and their zero-shot performance on different downstream NLP tasks. In this work, we show that this phenomenon does not hold when evaluating large LMs on tasks with negated prompts, but instead shows an inverse scaling law. We evaluate 9 different tasks with negated prompts on (1) pretrained LMs (OPT & GPT-3) of varying sizes (125M - 175B), (2) LMs further pretrained to generalize to novel prompts (InstructGPT), (3) LMs provided with few-shot examples, and (4) LMs fine-tuned specifically on negated prompts; all LM types perform worse on negated prompts as they scale and show a huge performance gap between the human performance when comparing the average score on both original and negated prompts. By highlighting a critical limitation of existing LMs and methods, we urge the community to develop new approaches of developing LMs that actually follow the given instructions. We provide the code and the datasets to explore negated prompts at https://github.com/joeljang/negated-prompts-for-llms