# A Symbolic Framework for Evaluating Mathematical Reasoning and Generalisation with Transformers arXiv: https://arxiv.org/abs/2305.12563 ยท HTML: https://mlsanity.com/paper/2305.12563/ ## Key facts - Cited by 15 papers in the map (data through 14 Oct 2026). - Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. - Builds on 30 of its 69 references that are in the map. - First posted on arXiv 21 May 2023. ## Placed in - LLM reasoning / [Mathematical reasoning](https://mlsanity.com/lr/mathematical-reasoning/) ## Builds on (most cited first, 30 of 30) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021) - [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023) - [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019) - [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019) - [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022) - [Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks](https://mlsanity.com/paper/2211.12588/) (2022) - [Are NLP Models really able to Solve Simple Math Word Problems?](https://mlsanity.com/paper/2103.07191/) (2021) - [Learning the Difference that Makes a Difference with Counterfactually-Augmented Data](https://mlsanity.com/paper/1909.12434/) (2019) - [Generative Language Modeling for Automated Theorem Proving](https://mlsanity.com/paper/2009.03393/) (2020) - [Analysing Mathematical Reasoning Abilities of Neural Models](https://mlsanity.com/paper/1904.01557/) (2019) - [Deep Learning for Symbolic Mathematics](https://mlsanity.com/paper/1912.01412/) (2019) - [Transformers as Soft Reasoners over Language](https://mlsanity.com/paper/2002.05867/) (2020) - [Mathematical Capabilities of ChatGPT](https://mlsanity.com/paper/2301.13867/) (2023) - [On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines](https://mlsanity.com/paper/2006.04884/) (2020) - [Will we run out of data? Limits of LLM scaling based on human-generated data](https://mlsanity.com/paper/2211.04325/) (2022) - [A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Models](https://mlsanity.com/paper/2210.12023/) (2022) - [MathBERT: A Pre-Trained Model for Mathematical Formula Understanding](https://mlsanity.com/paper/2105.00377/) (2021) - [INT: An Inequality Benchmark for Evaluating Generalization in Theorem Proving](https://mlsanity.com/paper/2007.02924/) (2020) - [MathBERT: A Pre-trained Language Model for General NLP Tasks in Mathematics Education](https://mlsanity.com/paper/2106.07340/) (2021) - [The Use of Deep Learning for Symbolic Integration: A Review of (Lample and Charton, 2019)](https://mlsanity.com/paper/1912.05752/) (2019) - [BERT is not The Count: Learning to Match Mathematical Statements with Proofs](https://mlsanity.com/paper/2302.09350/) (2023) - [Evaluating Token-Level and Passage-Level Dense Retrieval Models for Math Information Retrieval](https://mlsanity.com/paper/2203.11163/) (2022) - [IsarStep: a Benchmark for High-level Mathematical Reasoning](https://mlsanity.com/paper/2006.09265/) (2020) - [Enhancing Neural Mathematical Reasoning by Abductive Combination with Symbolic Library](https://mlsanity.com/paper/2203.14487/) (2022) - [Algebra Error Classification with Large Language Models](https://mlsanity.com/paper/2305.06163/) (2023) - [Mathematical Reasoning via Self-supervised Skip-tree Training](https://mlsanity.com/paper/2006.04757/) (2020) - [PhysNLU: A Language Resource for Evaluating Natural Language Understanding and Explanation Coherence in Physics](https://mlsanity.com/paper/2201.04275/) (2022) - [Estimating the Causal Effects of Natural Logic Features in Neural NLI Models](https://mlsanity.com/paper/2305.08572/) (2023) ## Built on by (most cited first, 15 of 15) - [Autoformalization in the Wild: Assessing LLMs on Real-World Mathematical Definitions](https://mlsanity.com/paper/2502.12065/) (2025) - [Evaluating LLMs' Mathematical and Coding Competency through Ontology-guided Interventions](https://mlsanity.com/paper/2401.09395/) (2024) - [CoMAT: Chain of Mathematically Annotated Thought Improves Mathematical Reasoning](https://mlsanity.com/paper/2410.10336/) (2024) - [A Theory for Length Generalization in Learning to Reason](https://mlsanity.com/paper/2404.00560/) (2024) - [Conditions for Length Generalization in Learning Reasoning Skills](https://mlsanity.com/paper/2311.16173/) (2023) - [Multi-Operational Mathematical Derivations in Latent Space](https://mlsanity.com/paper/2311.01230/) (2023) - [Improving Semantic Control in Discrete Latent Spaces with Transformer Quantized Variational Autoencoders](https://mlsanity.com/paper/2402.00723/) (2024) - [MASA: LLM-Driven Multi-Agent Systems for Autoformalization](https://mlsanity.com/paper/2510.08988/) (2025) - [Integrating Expert Knowledge into Logical Programs via LLMs](https://mlsanity.com/paper/2502.12275/) (2025) - [RoMath: A Mathematical Reasoning Benchmark in Romanian](https://mlsanity.com/paper/2409.11074/) (2024) - [Learning to Disentangle Latent Reasoning Rules with Language VAEs: A Systematic Study](https://mlsanity.com/paper/2506.19418/) (2025) - [Controlling Equational Reasoning in Large Language Models with Prompt Interventions](https://mlsanity.com/paper/2307.09998/) (2023) - [MAMUT: A Novel Framework for Modifying Mathematical Formulas for the Generation of Specialized Datasets for Language Model Training](https://mlsanity.com/paper/2502.20855/) (2025) - [Bridging Compositional and Distributional Semantics: A Survey on Latent Semantic Geometry via AutoEncoder](https://mlsanity.com/paper/2506.20083/) (2025) - [When Tokenization is Secretly Output Supervision](https://mlsanity.com/paper/2609.01386/) (2026) ## Abstract (arXiv) This paper proposes a methodology for generating and perturbing detailed derivations of equations at scale, aided by a symbolic engine, to evaluate the generalisability of Transformers to out-of-distribution mathematical reasoning problems. Instantiating the framework in the context of sequence classification tasks, we compare the capabilities of GPT-4, GPT-3.5, and a canon of fine-tuned BERT models, exploring the relationship between specific operators and generalisation failure via the perturbation of reasoning aspects such as symmetry and variable surface forms. Surprisingly, our empirical evaluation reveals that the average in-distribution performance of fine-tuned models surpasses GPT-3.5, and rivals GPT-4. However, perturbations to input reasoning can reduce their performance by up to 80 F1 points. Overall, the results suggest that the in-distribution performance of smaller open-source models may potentially rival GPT by incorporating appropriately structured derivation dependencies during training, and highlight a shared weakness between BERT and GPT involving a relative inability to decode indirect references to mathematical entities. We release the full codebase, constructed datasets, and fine-tuned models to encourage future progress in the field.