# ScoNe: Benchmarking Negation Reasoning in Language Models With Fine-Tuning and In-Context Learning

arXiv: https://arxiv.org/abs/2305.19426 · HTML: https://mlsanity.com/paper/2305.19426/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 7 of its 42 references that are in the map.
- First posted on arXiv 30 May 2023.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 7 of 7)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Causal Abstractions of Neural Networks](https://mlsanity.com/paper/2106.02997/) (2021)
- [Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations](https://mlsanity.com/paper/2303.02536/) (2023)
- [CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation](https://mlsanity.com/paper/2211.00295/) (2022)
- [Causal Proxy Models for Concept-Based Model Explanations](https://mlsanity.com/paper/2209.14279/) (2022)
- [Inducing Character-level Structure in Subword-based Language Models with Type-level Interchange Intervention Training](https://mlsanity.com/paper/2212.09897/) (2022)
- [Non-entailed subsequences as a challenge for natural language inference](https://mlsanity.com/paper/1811.12112/) (2018)

## Built on by (most cited first, 18 of 18)

- [Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs](https://mlsanity.com/paper/2406.11695/) (2024)
- [FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets](https://mlsanity.com/paper/2307.10928/) (2023)
- [A Systematic Survey of Automatic Prompt Optimization Techniques](https://mlsanity.com/paper/2502.16923/) (2025)
- [Are Multilingual LLMs Culturally-Diverse Reasoners? An Investigation into Multicultural Proverbs and Sayings](https://mlsanity.com/paper/2309.08591/) (2023)
- [Can Language Models Be Tricked by Language Illusions? Easier with Syntax, Harder with Semantics](https://mlsanity.com/paper/2311.01386/) (2023)
- [A Theory for Length Generalization in Learning to Reason](https://mlsanity.com/paper/2404.00560/) (2024)
- [Conditions for Length Generalization in Learning Reasoning Skills](https://mlsanity.com/paper/2311.16173/) (2023)
- [Thunder-NUBench: A Benchmark for LLMs' Sentence-Level Negation Understanding](https://mlsanity.com/paper/2506.14397/) (2025)
- [Revisiting subword tokenization: A case study on affixal negation in large language models](https://mlsanity.com/paper/2404.02421/) (2024)
- [LangProBe: a Language Programs Benchmark](https://mlsanity.com/paper/2502.20315/) (2025)
- [Propositional Logic for Probing Generalization in Neural Networks](https://mlsanity.com/paper/2506.08978/) (2025)
- [From Triage to Discharge: A Survey of NLP Tasks, Methods, and Open Challenges in the Emergency Department](https://mlsanity.com/paper/2608.23627/) (2026)
- [Can Large Language Models Robustly Perform Natural Language Inference for Japanese Comparatives?](https://mlsanity.com/paper/2509.13695/) (2025)
- [Implementing a Logical Inference System for Japanese Comparatives](https://mlsanity.com/paper/2509.13734/) (2025)
- [Beyond Understanding: Evaluating the Pragmatic Gap in LLMs' Cultural Processing of Figurative Language](https://mlsanity.com/paper/2510.23828/) (2025)
- [Which bird does not have wings: Negative-constrained KGQA with Schema-guided Semantic Matching and Self-directed Refinement](https://mlsanity.com/paper/2604.14749/) (2026)
- [Not all Negation Cues are Equal: Affixal Negations Yield Better Negation Understanding](https://mlsanity.com/paper/2609.13685/) (2026)
- [Reading Right, Acting Wrong: Diagnosing Decision Failures of System One Models in Agent Loops](https://mlsanity.com/paper/2610.18579/) (2026)

## Abstract (arXiv)

A number of recent benchmarks seek to assess how well models handle natural language negation. However, these benchmarks lack the controlled example paradigms that would allow us to infer whether a model had learned how negation morphemes semantically scope. To fill these analytical gaps, we present the Scoped Negation NLI (ScoNe-NLI) benchmark, which contains contrast sets of six examples with up to two negations where either zero, one, or both negative morphemes affect the NLI label. We use ScoNe-NLI to assess fine-tuning and in-context learning strategies. We find that RoBERTa and DeBERTa models solve ScoNe-NLI after many shot fine-tuning. For in-context learning, we test InstructGPT models and find that most prompt strategies are not successful, including those using step-by-step reasoning. To better understand this result, we extend ScoNe with ScoNe-NLG, a sentence completion test set that embeds negation reasoning in short narratives. Here, InstructGPT is successful, which reveals the model can correctly reason about negation, but struggles to do so on prompt-adapted NLI examples outside of its core pretraining regime.