mlsanityData to 7 Oct 2026

Measuring Systematic Generalization in Neural Proof Generation with Transformers

arXiv 2009.14786 ↗First posted 30 Sep 2020cs.LG cs.AI cs.CLcited by 43

Not on the map. Why?

Abstract (arXiv)

We are interested in understanding how well Transformer language models (TLMs) can perform reasoning tasks when trained on knowledge encoded in the form of natural language. We investigate their systematic generalization abilities on a logical reasoning task in natural language, which involves reasoning over relationships between entities grounded in first-order logical proofs. Specifically, we perform soft theorem-proving by leveraging TLMs to generate natural language proofs. We test the generated proofs for logical consistency, along with the accuracy of the final inference. We observe length-generalization issues when evaluated on longer-than-trained sequences. However, we observe TLMs improve their generalization performance after being exposed to longer, exhaustive proofs. In addition, we discover that TLMs are able to generalize better using backward-chaining proofs compared to their forward-chaining counterparts, while they find it easier to generate forward chaining proofs. We observe that models that are not trained to generate proofs are better at generalizing to problems based on longer proofs. This suggests that Transformers have efficient internal reasoning strategies that are harder to interpret. These results highlight the systematic generalization behavior of TLMs in the context of logical reasoning, and we believe this work motivates deeper inspection of their underlying reasoning strategies.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). First posted on arXiv 30 Sep 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 13; 2022: 18; 2023: 6; 2024: 4; 2025: 2.

43 papers.

Sort:

By area:

  1. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  2. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  3. 2022A Review on Language Models as Knowledge Basescited by 156
  4. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  5. 2022On the Paradox of Learning to Reason from Datacited by 93
  6. 2023Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examplescited by 84
  7. 2022LAMBADA: Backward Chaining for Automated Reasoning in Natural Languagecited by 66
  8. 2022Generating Natural Language Proofs with Verifier-Guided Searchcited by 52
  9. 2021Systematic Generalization with Edge Transformerscited by 38
  10. 2022The Neural Race Reduction: Dynamics of Abstraction in Gated Networkscited by 37
  11. 2024Enhancing Reasoning Capabilities of LLMs via Principled Synthetic Logic Corpuscited by 29
  12. 2022RobustLR: Evaluating Robustness to Logical Perturbation in Deductive Reasoningcited by 22
31 more
  1. 2024Position: Understanding LLMs Requires More Than Statistical Generalizationcited by 20
  2. 2021Pushing the Limits of Rule Reasoning in Transformers through Natural Language Satisfiabilitycited by 18
  3. 2023Learning Deductive Reasoning from Synthetic Corpus based on Formal Logiccited by 18
  4. 2022AbductionRules: Training Transformers to Explain Unexpected Inputscited by 17
  5. 2023RECKONING: Reasoning through Dynamic Knowledge Encodingcited by 17
  6. 2021Thinking Aloud: Dynamic Context Generation Improves Zero-Shot Reasoning Performance of GPT-2cited by 15
  7. 2021multiPRover: Generating Multiple Proofs for Improved Interpretability in Rule Reasoningcited by 15
  8. 2024A Survey on Compositional Learning of AI Models: Theoretical and Experimental Practicescited by 15
  9. 2021NOAHQA: Numerical Reasoning with Interpretable Graph Question Answering Datasetcited by 14
  10. 2022Summarization Programs: Interpretable Abstractive Summarization with Neural Modular Treescited by 14
  11. 2021Learning Symbolic Rules for Reasoning in Quasi-Natural Languagecited by 9
  12. 2024A Survey on Symbolic Knowledge Distillation of Large Language Modelscited by 9
  13. 2021Does Pre-training Induce Systematic Inference? How Masked Language Models Acquire Commonsense Knowledgecited by 8
  14. 2021Diff-Explainer: Differentiable Convex Optimization for Explainable Multi-hop Inferencecited by 7
  15. 2021DeepA2: A Modular Framework for Deep Argument Analysis with Pretrained Neural Text2Text Language Modelscited by 7
  16. 2021Dyna-bAbI: unlocking bAbI's potential with dynamic synthetic benchmarkingcited by 7
  17. 2025Random Scaling of Emergent Capabilitiescited by 7
  18. 2022Breakpoint Transformers for Modeling and Tracking Intermediate Beliefscited by 6
  19. 2022Transformer Module Networks for Systematic Generalization in Visual Question Answeringcited by 5
  20. 2022Learning to Reason With Relational Abstractionscited by 5
  21. 2025Are Language Models Efficient Reasoners? A Perspective from Logic Programmingcited by 4
  22. 2021Leveraging Transformers for StarCraft Macromanagement Predictioncited by 3
  23. 2022Does Entity Abstraction Help Generative Transformers Reason?cited by 3
  24. 2022Logically Consistent Adversarial Attacks for Soft Theorem Proverscited by 2
  25. 2022LogicRank: Logic Induced Reranking for Generative Text-to-Image Systemscited by 2
  26. 2023D3: Data Diversity Design for Systematic Generalization in Visual Question Answeringcited by 2
  27. 2021Enhancing the Transformer Decoder with Transition-based Syntaxcited by 1
  28. 2022ChemAlgebra: Algebraic Reasoning on Chemical Reactionscited by 1
  29. 2023Empirical Investigation of Neural Symbolic Reasoning Strategiescited by 1
  30. 2021Systematic Generalization in Neural Networks-based Multivariate Time Series Forecasting Modelscited by 0
  31. 2022Generating Intermediate Steps for NLI with Next-Step Supervisioncited by 0