# Transformers to Predict the Applicability of Symbolic Integration Routines

arXiv: https://arxiv.org/abs/2410.23948 · HTML: https://mlsanity.com/paper/2410.23948/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 21 references that are in the map.
- First posted on arXiv 31 Oct 2024.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Captum: A unified and generic model interpretability library for PyTorch](https://mlsanity.com/paper/2009.07896/) (2020)
- [Deep Learning for Symbolic Mathematics](https://mlsanity.com/paper/1912.01412/) (2019)
- [What Algorithms can Transformers Learn? A Study in Length Generalization](https://mlsanity.com/paper/2310.16028/) (2023)
- [A Neural Network Solves, Explains, and Generates University Math Problems by Program Synthesis and Few-Shot Learning at Human Level](https://mlsanity.com/paper/2112.15594/) (2021)
- [Learning the greatest common divisor: explaining transformer predictions](https://mlsanity.com/paper/2308.15594/) (2023)
- [Pretrained Language Models are Symbolic Mathematics Solvers too!](https://mlsanity.com/paper/2110.03501/) (2021)
- [Can Neural Networks Learn Symbolic Rewriting?](https://mlsanity.com/paper/1911.04873/) (2019)
- [The Use of Deep Learning for Symbolic Integration: A Review of (Lample and Charton, 2019)](https://mlsanity.com/paper/1912.05752/) (2019)
- [Machine learning detects terminal singularities](https://mlsanity.com/paper/2310.20458/) (2023)
- [Explainable AI Insights for Symbolic Computation: A case study on selecting the variable ordering for cylindrical algebraic decomposition](https://mlsanity.com/paper/2304.12154/) (2023)
- [Generating Elementary Integrable Expressions](https://mlsanity.com/paper/2306.15572/) (2023)
- [Symbolic Integration Algorithm Selection with Machine Learning: LSTMs vs Tree LSTMs](https://mlsanity.com/paper/2404.14973/) (2024)
- [The Liouville Generator for Producing Integrable Expressions](https://mlsanity.com/paper/2406.11631/) (2024)

## Built on by (most cited first, 2 of 2)

- [Studying number theory with deep learning: a case study with the M"obius and squarefree indicator functions](https://mlsanity.com/paper/2502.10335/) (2025)
- [Tree-Based Deep Learning for Ranking Symbolic Integration Algorithms](https://mlsanity.com/paper/2508.06383/) (2025)

## Abstract (arXiv)

Symbolic integration is a fundamental problem in mathematics: we consider how machine learning may be used to optimise this task in a Computer Algebra System (CAS). We train transformers that predict whether a particular integration method will be successful, and compare against the existing human-made heuristics (called guards) that perform this task in a leading CAS. We find the transformer can outperform these guards, gaining up to 30% accuracy and 70% precision. We further show that the inference time of the transformer is inconsequential which shows that it is well-suited to include as a guard in a CAS. Furthermore, we use Layer Integrated Gradients to interpret the decisions that the transformer is making. If guided by a subject-matter expert, the technique can explain some of the predictions based on the input tokens, which can lead to further optimisations.