mlsanityData to 14 Oct 2026

Assessing Logical Reasoning Capabilities of Encoder-Only Transformer Models

arXiv 2312.11720 ↗First posted 18 Dec 2023cs.CL cs.AIcited by 2

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Logical reasoning is central to complex human activities, such as thinking, debating, and planning; it is also a central component of many AI systems as well. In this paper, we investigate the extent to which encoder-only transformer language models (LMs) can reason according to logical rules. We ask whether those LMs can deduce theorems in propositional calculus and first-order logic; if their relative success in these problems reflects general logical capabilities; and which layers contribute the most to the task. First, we show for several encoder-only LMs that they can be trained, to a reasonable degree, to determine logical validity on various datasets. Next, by cross-probing fine-tuned models on these datasets, we show that LMs have difficulty in transferring their putative logical reasoning ability, which suggests that they may have learned dataset-specific features, instead of a general capability. Finally, we conduct a layerwise probing experiment, which shows that the hypothesis classification task is mostly solved through higher layers.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 47 of its 58 references that are in the map. First posted on arXiv 18 Dec 2023.

Builds on

47 of 58 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners24 May
  2. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought3 Oct
  3. 2022FOLIO: Natural Language Reasoning with First-Order Logic2 Sep
  4. 2022Language models show human-like content effects on reasoning tasks14 Jul
  5. 2022On the Paradox of Learning to Reason from Data23 May

Other references · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  8. 2020Longformer: The Long-Document Transformercited by 3,413
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  10. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  11. 2022Emergent Abilities of Large Language Modelscited by 2,512
  12. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
30 more
  1. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  2. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  3. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  4. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  5. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  6. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  7. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  8. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
  9. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  10. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  11. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 503
  12. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 457
  13. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 438
  14. 2023Language Models Represent Space and Timecited by 316
  15. 2020Transformers as Soft Reasoners over Languagecited by 270
  16. 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 243
  17. 2020ReClor: A Reading Comprehension Dataset Requiring Logical Reasoningcited by 217
  18. 2019CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Textcited by 162
  19. 2020When Do You Need Billions of Words of Pretraining Data?cited by 104
  20. 2019Counterfactual Story Reasoning and Generationcited by 88
  21. 2021Probing Across Time: What Does RoBERTa Know and When?cited by 73
  22. 2019Visualizing and Understanding the Effectiveness of BERTcited by 67
  23. 2021The paradox of the compositionality of natural language: a neural machine translation case studycited by 59
  24. 2020Teaching Temporal Logics to Neural Networkscited by 51
  25. 2021I Wish I Would Have Loved This One, But I Didn't -- A Multilingual Dataset for Counterfactual Detection in Product Reviewscited by 45
  26. 2020Infusing Finetuning with Semantic Dependenciescited by 18
  27. 2020SemEval-2020 Task 5: Counterfactual Recognitioncited by 12
  28. 2022Negation, Coordination, and Quantifiers in Contextualized Language Modelscited by 9
  29. 2022Are Representations Built from the Ground Up? An Empirical Examination of Local Composition in Language Modelscited by 9
  30. 2022Logically Consistent Adversarial Attacks for Soft Theorem Proverscited by 2

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Surveycited by 83
  2. 2025Causal Reasoning Favors Encoders: On The Limits of Decoder-Only Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Reasoning in Transformers -- Mitigating Spurious Correlations and Reasoning Shortcuts
  2. 2023Transformers in the Service of Description Logic-based Contexts
  3. 2024Transformer-based Language Models for Reasoning in the Description Logic ALCQ
  4. 2025Logical Reasoning in Large Language Models: A Survey
  5. 2023Logical Reasoning over Natural Language as Knowledge Representation: A Survey
  6. 2026To See the Unseen: on the Generalization Ability of Transformers in Symbolic Reasoning
  7. 2023Examining the Emergence of Deductive Reasoning in Generative Language Models
  8. 2022Can Transformers Reason in Fragments of Natural Language?