mlsanityData to 7 Oct 2026

Assessing Step-by-Step Reasoning against Lexical Negation: A Case Study on Syllogism

arXiv 2310.14868 ↗First posted 23 Oct 2023cs.CLcited by 6

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large language models (LLMs) take advantage of step-by-step reasoning instructions, e.g., chain-of-thought (CoT) prompting. Building on this, their ability to perform CoT-style reasoning robustly is of interest from a probing perspective. In this study, we inspect the step-by-step reasoning ability of LLMs with a focus on negation, which is a core linguistic phenomenon that is difficult to process. In particular, we introduce several controlled settings (e.g., reasoning in case of fictional entities) to evaluate the logical reasoning abilities of the models. We observed that dozens of modern LLMs were not robust against lexical negation (e.g., plausible ->implausible) when performing CoT-style reasoning, and the results highlight unique limitations in each LLM family.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 29 of its 34 references that are in the map. First posted on arXiv 23 Oct 2023.

Builds on

29 of 34 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023LLaMA: Open and Efficient Foundation Language Models27 Feb
  2. 2023Empirical Investigation of Neural Symbolic Reasoning Strategies16 Feb

Other references · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  7. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  8. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  9. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  10. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
  11. 2020Transformers as Soft Reasoners over Languagecited by 265
  12. 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 240
15 more
  1. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 235
  2. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 175
  3. 2020Unsupervised Commonsense Question Answering with Self-Talkcited by 161
  4. 2020NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraintscited by 146
  5. 2021Understanding by Understanding Not: Modeling Negation in Language Modelscited by 73
  6. 2023Language models are not naysayers: An analysis of language models on negation benchmarkscited by 73
  7. 2019Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIscited by 58
  8. 2022Probing for the Usage of Grammatical Numbercited by 57
  9. 2022Can Large Language Models Truly Understand Prompts? A Case Study with Negated Promptscited by 56
  10. 2019Probing What Different NLP Tasks Teach Machines about Function Word Comprehensioncited by 44
  11. 2020Critical Thinking for Language Modelscited by 31
  12. 2022Scaling Laws Under the Microscope: Predicting Transformer Performance from Small Scale Experimentscited by 23
  13. 2018Improving Machine Reading Comprehension with General Reading Strategiescited by 20
  14. 2021Think about it! Improving defeasible reasoning by first modeling the question scenariocited by 20
  15. 2023Do Deep Neural Networks Capture Compositionality in Arithmetic Reasoning?cited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 2; 2026: 2.

6 papers.

  1. 2023A Systematic Comparison of Syllogistic Reasoning in Humans and Language Modelscited by 20
  2. 2024Investigating and Addressing Hallucinations of LLMs in Tasks Involving Negationcited by 15
  3. 2025SR-FoT: A Syllogistic-Reasoning Framework of Thought for Large Language Models Tackling Knowledge-based Reasoning Taskscited by 3
  4. 2025The Impact of Negated Text on Hallucination with Large Language Modelscited by 0
  5. 2026Which bird does not have wings: Negative-constrained KGQA with Schema-guided Semantic Matching and Self-directed Refinementcited by 0
  6. 2026Not all Negation Cues are Equal: Affixal Negations Yield Better Negation Understandingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Chain-of-Thought Reasoning Without Prompting
  2. 2024Exploring Reasoning Biases in Large Language Models Through Syllogism: Insights from the NeuBAROCO Dataset
  3. 2025Negation: A Pink Elephant in the Large Language Models' Room?
  4. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters
  5. 2025Reasoning Capabilities and Invariability of Large Language Models
  6. 2024A Systematic Analysis of Large Language Models as Soft Reasoners: The Case of Syllogistic Inferences
  7. 2025Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
  8. 2024LLMs Faithfully and Iteratively Compute Answers During CoT: A Systematic Analysis With Multi-step Arithmetics