# Empirical Investigation of Neural Symbolic Reasoning Strategies

arXiv: https://arxiv.org/abs/2302.08148 · HTML: https://mlsanity.com/paper/2302.08148/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 27 of its 33 references that are in the map.
- First posted on arXiv 16 Feb 2023.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 27 of 27)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [Show Your Work: Scratchpads for Intermediate Computation with Language Models](https://mlsanity.com/paper/2112.00114/) (2021)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language](https://mlsanity.com/paper/2012.13048/) (2020)
- [Neurosymbolic AI: The 3rd Wave](https://mlsanity.com/paper/2012.05876/) (2020)
- [End-to-End Differentiable Proving](https://mlsanity.com/paper/1705.11040/) (2017)
- [Unsupervised Commonsense Question Answering with Self-Talk](https://mlsanity.com/paper/2004.05483/) (2020)
- [Making Transformers Solve Compositional Tasks](https://mlsanity.com/paper/2108.04378/) (2021)
- [What Makes Reading Comprehension Questions Easier?](https://mlsanity.com/paper/1808.09384/) (2018)
- [Generating Natural Language Proofs with Verifier-Guided Search](https://mlsanity.com/paper/2205.12443/) (2022)
- [BERT & Family Eat Word Salad: Experiments with Text Understanding](https://mlsanity.com/paper/2101.03453/) (2021)
- [Measuring Systematic Generalization in Neural Proof Generation with Transformers](https://mlsanity.com/paper/2009.14786/) (2020)
- [Is Neuro-Symbolic AI Meeting its Promise in Natural Language Processing? A Structured Review](https://mlsanity.com/paper/2202.12205/) (2022)
- [FaiRR: Faithful and Robust Deductive Reasoning over Natural Language](https://mlsanity.com/paper/2203.10261/) (2022)
- [Modeling Multi-hop Question Answering as Single Sequence Prediction](https://mlsanity.com/paper/2205.09226/) (2022)
- [Teaching Autoregressive Language Models Complex Tasks By Demonstration](https://mlsanity.com/paper/2109.02102/) (2021)
- [Numerical reasoning in machine reading comprehension tasks: are we there yet?](https://mlsanity.com/paper/2109.08207/) (2021)
- [DeepLogic: Towards End-to-End Differentiable Logical Reasoning](https://mlsanity.com/paper/1805.07433/) (2018)
- [Neural Unification for Logic Reasoning over Natural Language](https://mlsanity.com/paper/2109.08460/) (2021)

## Built on by (most cited first, 1 of 1)

- [Assessing Step-by-Step Reasoning against Lexical Negation: A Case Study on Syllogism](https://mlsanity.com/paper/2310.14868/) (2023)

## Abstract (arXiv)

Neural reasoning accuracy improves when generating intermediate reasoning steps. However, the source of this improvement is yet unclear. Here, we investigate and factorize the benefit of generating intermediate steps for symbolic reasoning. Specifically, we decompose the reasoning strategy w.r.t. step granularity and chaining strategy. With a purely symbolic numerical reasoning dataset (e.g., A=1, B=3, C=A+3, C?), we found that the choice of reasoning strategies significantly affects the performance, with the gap becoming even larger as the extrapolation length becomes longer. Surprisingly, we also found that certain configurations lead to nearly perfect performance, even in the case of length extrapolation. Our results indicate the importance of further exploring effective strategies for neural reasoning models.