# LogicInference: A New Dataset for Teaching Logical Inference to seq2seq Models

arXiv: https://arxiv.org/abs/2203.15099 · HTML: https://mlsanity.com/paper/2203.15099/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 25 references that are in the map.
- First posted on arXiv 28 Mar 2022.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [e-SNLI: Natural Language Inference with Natural Language Explanations](https://mlsanity.com/paper/1812.01193/) (2018)
- [Analysing Mathematical Reasoning Abilities of Neural Models](https://mlsanity.com/paper/1904.01557/) (2019)
- [Measuring Compositional Generalization: A Comprehensive Method on Realistic Data](https://mlsanity.com/paper/1912.09713/) (2019)
- [Compositionality decomposed: how do neural networks generalise?](https://mlsanity.com/paper/1908.08351/) (2019)
- [CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text](https://mlsanity.com/paper/1908.06177/) (2019)
- [Explaining Answers with Entailment Trees](https://mlsanity.com/paper/2104.08661/) (2021)
- [The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers](https://mlsanity.com/paper/2108.12284/) (2021)
- [Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architectures](https://mlsanity.com/paper/2007.08970/) (2020)
- [Making Transformers Solve Compositional Tasks](https://mlsanity.com/paper/2108.04378/) (2021)
- [Memorize or generalize? Searching for a compositional RNN in a haystack](https://mlsanity.com/paper/1802.06467/) (2018)

## Built on by (most cited first, 12 of 12)

- [Evaluating Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2310.19736/) (2023)
- [Natural Language Reasoning, A Survey](https://mlsanity.com/paper/2303.14725/) (2023)
- [LogiCoT: Logical Chain-of-Thought Instruction-Tuning](https://mlsanity.com/paper/2305.12147/) (2023)
- [Understanding the Language Model to Solve the Symbolic Multi-Step Reasoning Problem from the Perspective of Buffer Mechanism](https://mlsanity.com/paper/2405.15302/) (2024)
- [Reason from Fallacy: Enhancing Large Language Models' Logical Reasoning through Logical Fallacy Understanding](https://mlsanity.com/paper/2404.04293/) (2024)
- [Reasoning Bias of Next Token Prediction Training](https://mlsanity.com/paper/2502.02007/) (2025)
- [Logical forms complement probability in understanding language model (and human) performance](https://mlsanity.com/paper/2502.09589/) (2025)
- [LogicLearner: A Tool for the Guided Practice of Propositional Logic Proofs](https://mlsanity.com/paper/2503.19280/) (2025)
- [Transformers in the Service of Description Logic-based Contexts](https://mlsanity.com/paper/2311.08941/) (2023)
- [LogicPrpBank: A Corpus for Logical Implication and Equivalence](https://mlsanity.com/paper/2402.09609/) (2024)
- [Stress-Testing the Reasoning Competence of LLMs With Proofs Under Minimal Formalism](https://mlsanity.com/paper/2605.12524/) (2026)
- [ChLogic: Evaluating Robustness of Logical Reasoning in Chinese Expressions](https://mlsanity.com/paper/2606.17905/) (2026)

## Abstract (arXiv)

Machine learning models such as Transformers or LSTMs struggle with tasks that are compositional in nature such as those involving reasoning/inference. Although many datasets exist to evaluate compositional generalization, when it comes to evaluating inference abilities, options are more limited. This paper presents LogicInference, a new dataset to evaluate the ability of models to perform logical inference. The dataset focuses on inference using propositional logic and a small subset of first-order logic, represented both in semi-formal logical notation, as well as in natural language. We also report initial results using a collection of machine learning models to establish an initial baseline in this dataset.