# FaiRR: Faithful and Robust Deductive Reasoning over Natural Language

arXiv: https://arxiv.org/abs/2203.10261 · HTML: https://mlsanity.com/paper/2203.10261/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 28 of its 38 references that are in the map.
- First posted on arXiv 19 Mar 2022.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?](https://mlsanity.com/paper/2004.03685/) (2020)
- [ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language](https://mlsanity.com/paper/2012.13048/) (2020)
- [Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks](https://mlsanity.com/paper/1502.05698/) (2015)
- [Transformers as Soft Reasoners over Language](https://mlsanity.com/paper/2002.05867/) (2020)
- [Causality for Machine Learning](https://mlsanity.com/paper/1911.10500/) (2019)
- [Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias](https://mlsanity.com/paper/2004.12265/) (2020)
- [Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond](https://mlsanity.com/paper/2109.00725/) (2021)
- [CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text](https://mlsanity.com/paper/1908.06177/) (2019)
- [End-to-End Differentiable Proving](https://mlsanity.com/paper/1705.11040/) (2017)
- [Invariance, Causality and Robustness](https://mlsanity.com/paper/1812.08233/) (2018)
- [CausaLM: Causal Model Explanation Through Counterfactual Language Models](https://mlsanity.com/paper/2005.13407/) (2020)
- [QuaRTz: An Open-Domain Dataset of Qualitative Relationship Questions](https://mlsanity.com/paper/1909.03553/) (2019)
- [Reasoning Over Paragraph Effects in Situations](https://mlsanity.com/paper/1908.05852/) (2019)
- [Counterfactual Invariance to Spurious Correlations: Why and How to Pass Stress Tests](https://mlsanity.com/paper/2106.00545/) (2021)
- [Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMs](https://mlsanity.com/paper/1801.05453/) (2018)
- [A Survey on Semantic Parsing](https://mlsanity.com/paper/1812.00978/) (2018)
- [PRover: Proof Generation for Interpretable Reasoning over Rules](https://mlsanity.com/paper/2010.02830/) (2020)
- [Causal Reasoning for Algorithmic Fairness](https://mlsanity.com/paper/1805.05859/) (2018)
- [Discretized Integrated Gradients for Explaining Language Models](https://mlsanity.com/paper/2108.13654/) (2021)
- [NLProlog: Reasoning with Weak Unification for Question Answering in Natural Language](https://mlsanity.com/paper/1906.06187/) (2019)
- [multiPRover: Generating Multiple Proofs for Improved Interpretability in Rule Reasoning](https://mlsanity.com/paper/2106.01354/) (2021)

## Built on by (most cited first, 29 of 29)

- [Natural Language Reasoning, A Survey](https://mlsanity.com/paper/2303.14725/) (2023)
- [A Survey on Deep Learning for Theorem Proving](https://mlsanity.com/paper/2404.09939/) (2024)
- [Generating Natural Language Proofs with Verifier-Guided Search](https://mlsanity.com/paper/2205.12443/) (2022)
- [Explainable Claim Verification via Knowledge-Grounded Reasoning with Large Language Models](https://mlsanity.com/paper/2310.05253/) (2023)
- [How Interpretable are Reasoning Explanations from Prompting Large Language Models?](https://mlsanity.com/paper/2402.11863/) (2024)
- [Enhancing Reasoning Capabilities of LLMs via Principled Synthetic Logic Corpus](https://mlsanity.com/paper/2411.12498/) (2024)
- [RobustLR: Evaluating Robustness to Logical Perturbation in Deductive Reasoning](https://mlsanity.com/paper/2205.12598/) (2022)
- [Quantifying the Capabilities of LLMs across Scale and Precision](https://mlsanity.com/paper/2405.03146/) (2024)
- [RECKONING: Reasoning through Dynamic Knowledge Encoding](https://mlsanity.com/paper/2305.06349/) (2023)
- [Learning Deductive Reasoning from Synthetic Corpus based on Formal Logic](https://mlsanity.com/paper/2308.07336/) (2023)
- [Symbolic Working Memory Enhances Language Models for Complex Rule Application](https://mlsanity.com/paper/2408.13654/) (2024)
- [Faithful Question Answering with Monte-Carlo Planning](https://mlsanity.com/paper/2305.02556/) (2023)
- [Enhancing Ethical Explanations of Large Language Models through Iterative Symbolic Refinement](https://mlsanity.com/paper/2402.00745/) (2024)
- [Iterated Decomposition: Improving Science Q&A by Supervising Reasoning Processes](https://mlsanity.com/paper/2301.01751/) (2023)
- [GLoRE: Evaluating Logical Reasoning of Large Language Models](https://mlsanity.com/paper/2310.09107/) (2023)
- [Reverse Question Answering: Can an LLM Write a Question so Hard (or Bad) that it Can't Answer?](https://mlsanity.com/paper/2410.15512/) (2024)
- [RLET: A Reinforcement Learning Based Approach for Explainable QA with Entailment Trees](https://mlsanity.com/paper/2210.17095/) (2022)
- [Empower Nested Boolean Logic via Self-Supervised Curriculum Learning](https://mlsanity.com/paper/2310.05450/) (2023)
- [CLOMO: Counterfactual Logical Modification with Large Language Models](https://mlsanity.com/paper/2311.17438/) (2023)
- [A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data](https://mlsanity.com/paper/2601.17717/) (2026)
- [Can Pretrained Language Models (Yet) Reason Deductively?](https://mlsanity.com/paper/2210.06442/) (2022)
- [APOLLO: A Simple Approach for Adaptive Pretraining of Language Models for Logical Reasoning](https://mlsanity.com/paper/2212.09282/) (2022)
- [NL2LOGIC: AST-Guided Translation of Natural Language into First-Order Logic with Large Language Models](https://mlsanity.com/paper/2602.13237/) (2026)
- [Improving Arithmetic Reasoning Ability of Large Language Models through Relation Tuples, Verification and Dynamic Feedback](https://mlsanity.com/paper/2406.17873/) (2024)
- [SEER: Facilitating Structured Reasoning and Explanation via Reinforcement Learning](https://mlsanity.com/paper/2401.13246/) (2024)
- [RISCORE: Enhancing In-Context Riddle Solving in Language Models through Context-Reconstructed Example Augmentation](https://mlsanity.com/paper/2409.16383/) (2024)
- [Empirical Investigation of Neural Symbolic Reasoning Strategies](https://mlsanity.com/paper/2302.08148/) (2023)
- [Explainable Verbal Reasoner Plus (EVR+): A Natural Language Reasoning Framework that Supports Diverse Compositional Reasoning](https://mlsanity.com/paper/2305.00061/) (2023)
- [Automated Annotation of Evolving Corpora for Augmenting Longitudinal Network Data: A Framework Integrating Large Language Models and Expert Knowledge](https://mlsanity.com/paper/2503.01672/) (2025)

## Abstract (arXiv)

Transformers have been shown to be able to perform deductive reasoning on a logical rulebase containing rules and statements written in natural language. Recent works show that such models can also produce the reasoning steps (i.e., the proof graph) that emulate the model's logical reasoning process. Currently, these black-box models generate both the proof graph and intermediate inferences within the same model and thus may be unfaithful. In this work, we frame the deductive logical reasoning task by defining three modular components: rule selection, fact selection, and knowledge composition. The rule and fact selection steps select the candidate rule and facts to be used and then the knowledge composition combines them to generate new inferences. This ensures model faithfulness by assured causal relation from the proof step to the inference reasoning. To test our framework, we propose FaiRR (Faithful and Robust Reasoner) where the above three components are independently modeled by transformers. We observe that FaiRR is robust to novel language perturbations, and is faster at inference than previous works on existing reasoning datasets. Additionally, in contrast to black-box generative models, the errors made by FaiRR are more interpretable due to the modular approach.