mlsanityData to 14 Oct 2026

RobustLR: Evaluating Robustness to Logical Perturbation in Deductive Reasoning

arXiv 2205.12598 ↗First posted 25 May 2022cs.CL cs.LG cs.LOcited by 22

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Transformers have been shown to be able to perform deductive reasoning on a logical rulebase containing rules and statements written in English natural language. While the progress is promising, it is currently unclear if these models indeed perform logical reasoning by understanding the underlying logical semantics in the language. To this end, we propose RobustLR, a suite of evaluation datasets that evaluate the robustness of these models to minimal logical edits in rulebases and some standard logical equivalence conditions. In our experiments with RoBERTa and T5, we find that the models trained in prior works do not perform consistently on the different perturbations in RobustLR, thus showing that the models are not robust to the proposed logical perturbations. Further, we find that the models find it especially hard to learn logical negation and disjunction operators. Overall, using our evaluation sets, we demonstrate some shortcomings of the deductive reasoning-based language models, which can eventually help towards designing better models for logical reasoning over natural language. All the datasets and code base have been made publicly available.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 21 of its 27 references that are in the map. First posted on arXiv 25 May 2022.

Builds on

21 of 27 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022On the Paradox of Learning to Reason from Data23 May
  2. 2022FaiRR: Faithful and Robust Deductive Reasoning over Natural Language19 Mar

Other references · 19

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  5. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  6. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  7. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  8. 2018Annotation Artifacts in Natural Language Inference Datacited by 671
  9. 2019Neural Probabilistic Logic Programming in DeepProbLogcited by 394
  10. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 324
  11. 2020Transformers as Soft Reasoners over Languagecited by 270
  12. 2020ReClor: A Reading Comprehension Dataset Requiring Logical Reasoningcited by 217
7 more
  1. 2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
  2. 2019CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Textcited by 162
  3. 2017End-to-End Differentiable Provingcited by 161
  4. 2020MuTual: A Dataset for Multi-Turn Dialogue Reasoningcited by 115
  5. 2019Understanding Dataset Design Choices for Multi-hop Reasoningcited by 50
  6. 2020Measuring Systematic Generalization in Neural Proof Generation with Transformerscited by 43
  7. 2020RICA: Evaluating Robust Inference Capabilities Based on Commonsense Axiomscited by 28

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 6; 2025: 5; 2026: 2.

22 papers.

Sort:

By area:

  1. 2023Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debatecited by 94
  2. 2023Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examplescited by 84
  3. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Surveycited by 83
  4. 2023Logical Reasoning over Natural Language as Knowledge Representation: A Surveycited by 32
  5. 2024Evaluating LLMs' Mathematical and Coding Competency through Ontology-guided Interventionscited by 20
  6. 2023RECKONING: Reasoning through Dynamic Knowledge Encodingcited by 18
  7. 2024Exploring the Compositional Deficiency of Large Language Models in Mathematical Reasoningcited by 15
  8. 2024Are Machines Better at Complex Reasoning? Unveiling Human-Machine Inference Gaps in Entailment Verificationcited by 13
  9. 2023Assessing and Enhancing the Robustness of Large Language Models with Task Structure Variations for Logical Reasoningcited by 10
  10. 2023On Robustness of Finetuned Transformer-based NLP Modelscited by 8
  11. 2023Empower Nested Boolean Logic via Self-Supervised Curriculum Learningcited by 7
  12. 2025Evaluating the Logical Reasoning Abilities of Large Reasoning Modelscited by 7
10 more
  1. 2025Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directionscited by 5
  2. 2023Breaking the Language Barrier: Improving Cross-Lingual Reasoning with Structured Self-Attentioncited by 4
  3. 2023Benchmarking Robustness of Text-Image Composed Retrievalcited by 2
  4. 2024IndicSentEval: How Effectively do Multilingual Transformer Models encode Linguistic Properties for Indic Languages?cited by 2
  5. 2025LogicTree: Structured Proof Exploration for Coherent and Rigorous Logical Reasoning with Large Language Modelscited by 1
  6. 2025Is Large Language Model Performance on Reasoning Tasks Impacted by Different Ways Questions Are Asked?cited by 1
  7. 2024Enhancing Robustness in Biomedical NLI Models: A Probing Approach for Clinical Trialscited by 0
  8. 2025Causal Reasoning Favors Encoders: On The Limits of Decoder-Only Modelscited by 0
  9. 2026NormWorlds-CF: Solver-Verified Counterfactual Normative Reasoning with Metamorphic-Relation GRPOcited by 0
  10. 2026Hob-VL: A Benchmark for Visually Grounded Boolean Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Investigating the Robustness of Deductive Reasoning with Large Language Models
  2. 2024Transformer-based Language Models for Reasoning in the Description Logic ALCQ
  3. 2025JustLogic: A Comprehensive Benchmark for Evaluating Deductive Reasoning in Large Language Models
  4. 2023Learning Deductive Reasoning from Synthetic Corpus based on Formal Logic
  5. 2024Scaling Synthetic Logical Reasoning Datasets with Context-Sensitive Declarative Grammars
  6. 2023Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond
  7. 2025Can Large Language Models Learn Formal Logic? A Data-Driven Training and Evaluation Framework
  8. 2025Negation: A Pink Elephant in the Large Language Models' Room?