mlsanityData to 7 Oct 2026

Can Transformers Reason in Fragments of Natural Language?

arXiv 2211.05417 ↗First posted 10 Nov 2022cs.CL cs.AIcited by 8

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

State-of-the-art deep-learning-based approaches to Natural Language Processing (NLP) are credited with various capabilities that involve reasoning with natural language texts. In this paper we carry out a large-scale empirical study investigating the detection of formally valid inferences in controlled fragments of natural language for which the satisfiability problem becomes increasingly complex. We find that, while transformer-based language models perform surprisingly well in these scenarios, a deeper analysis re-veals that they appear to overfit to superficial patterns in the data rather than acquiring the logical principles governing the reasoning in these fragments.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 27 of its 49 references that are in the map. First posted on arXiv 10 Nov 2022.

Builds on

27 of 49 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  9. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  10. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  11. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  12. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
15 more
  1. 2015Neural Module Networkscited by 375
  2. 2020Transformers as Soft Reasoners over Languagecited by 265
  3. 2016Enhanced LSTM for Natural Language Inferencecited by 199
  4. 2018Semidefinite relaxations for certifying robustness to adversarial examplescited by 178
  5. 2019Certified Robustness to Adversarial Word Substitutionscited by 164
  6. 2018Breaking NLI Systems with Sentences that Require Simple Lexical Inferencescited by 134
  7. 2020Robustness Verification for Transformerscited by 76
  8. 2019Probing Natural Language Inference Models through Semantic Fragmentscited by 64
  9. 2018What Makes Reading Comprehension Questions Easier?cited by 59
  10. 2020Probing Linguistic Systematicitycited by 37
  11. 2020Do Neural Models Learn Systematicity of Monotonicity Inference in Natural Language?cited by 29
  12. 2020Leap-Of-Thought: Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledgecited by 24
  13. 2019Diversify Your Datasets: Analyzing Generalization via Controlled Variance in Adversarial Datasetscited by 21
  14. 2020A Framework for Evaluation of Machine Reading Comprehension Gold Standardscited by 18
  15. 2021Pushing the Limits of Rule Reasoning in Transformers through Natural Language Satisfiabilitycited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 3; 2024: 0; 2025: 2; 2026: 2.

8 papers.

  1. 2025ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoningcited by 186
  2. 2022LAMBADA: Backward Chaining for Automated Reasoning in Natural Languagecited by 66
  3. 2023Dialogue Chain-of-Thought Distillation for Commonsense-aware Conversational Agentscited by 41
  4. 2023Evaluating Large Language Models with NeuBAROCO: Syllogistic Reasoning Ability and Human-like Biasescited by 14
  5. 2025Investigating Language Model Capabilities to Represent and Process Formal Knowledge: A Preliminary Study to Assist Ontology Engineeringcited by 2
  6. 2026Invariant Learning Dynamics of Transformers in Inductive Reasoning Taskscited by 1
  7. 2023Transformers in the Service of Description Logic-based Contextscited by 0
  8. 2026Logical Judgments Under Pressure: Diagnosing Syllogistic Stability with Learned Soft Prefixescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Natural Language Satisfiability: Exploring the Problem Distribution and Evaluating Transformer-based Language Models
  2. 2024Reasoning in Transformers -- Mitigating Spurious Correlations and Reasoning Shortcuts
  3. 2024Transformers meet Neural Algorithmic Reasoners
  4. 2023Assessing Logical Reasoning Capabilities of Encoder-Only Transformer Models
  5. 2024Transformer-based Language Models for Reasoning in the Description Logic ALCQ
  6. 2023Logical Reasoning over Natural Language as Knowledge Representation: A Survey
  7. 2024Reliable Reasoning Beyond Natural Language
  8. 2023Large Language Models Are Not Strong Abstract Reasoners