# A Transformer Model for Symbolic Regression towards Scientific Discovery

arXiv: https://arxiv.org/abs/2312.04070 · HTML: https://mlsanity.com/paper/2312.04070/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 19 references that are in the map.
- First posted on arXiv 7 Dec 2023.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [AI Feynman: a Physics-Inspired Method for Symbolic Regression](https://mlsanity.com/paper/1905.11481/) (2019)
- [Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients](https://mlsanity.com/paper/1912.04871/) (2019)
- [Contemporary Symbolic Regression Methods and their Relative Performance](https://mlsanity.com/paper/2107.14351/) (2021)
- [End-to-end symbolic regression with transformers](https://mlsanity.com/paper/2204.10532/) (2022)
- [Neural Symbolic Regression that Scales](https://mlsanity.com/paper/2106.06427/) (2021)
- [SymbolicGPT: A Generative Transformer Model for Symbolic Regression](https://mlsanity.com/paper/2106.14131/) (2021)
- [Integration of Neural Network-Based Symbolic Regression in Deep Learning for Scientific Discovery](https://mlsanity.com/paper/1912.04825/) (2019)
- [Rethinking Symbolic Regression Datasets and Benchmarks for Scientific Discovery](https://mlsanity.com/paper/2206.10540/) (2022)
- [AI-Assisted Discovery of Quantitative and Formal Models in Social Science](https://mlsanity.com/paper/2210.00563/) (2022)

## Built on by (most cited first, 9 of 9)

- [Analyzing Generalization in Pre-Trained Symbolic Regression](https://mlsanity.com/paper/2509.19849/) (2025)
- [Generalizing the SINDy approach with nested neural networks](https://mlsanity.com/paper/2404.15742/) (2024)
- [Can Test-time Computation Mitigate Reproduction Bias in Neural Symbolic Regression?](https://mlsanity.com/paper/2505.22081/) (2025)
- [Towards symbolic regression for interpretable clinical decision scores](https://mlsanity.com/paper/2512.07961/) (2025)
- [How are Scientific Concepts Birthed? Typing Rules of Concept Formation in Theoretical Physics Reasoning](https://mlsanity.com/paper/2509.10740/) (2025)
- [Beyond the Training Distribution: Mapping Generalization Boundaries in Neural Program Synthesis](https://mlsanity.com/paper/2604.27551/) (2026)
- [EditSR: Enhancing Neural Symbolic Regression via Edit-based Rectification](https://mlsanity.com/paper/2606.07915/) (2026)
- [ERBench: A Benchmark and Testsuite for Equation Discovery Algorithms](https://mlsanity.com/paper/2606.09276/) (2026)
- [Synthesizing Physics Formulae with Transformers](https://mlsanity.com/paper/2610.03947/) (2026)

## Abstract (arXiv)

Symbolic Regression (SR) searches for mathematical expressions which best describe numerical datasets. This allows to circumvent interpretation issues inherent to artificial neural networks, but SR algorithms are often computationally expensive. This work proposes a new Transformer model aiming at Symbolic Regression particularly focused on its application for Scientific Discovery. We propose three encoder architectures with increasing flexibility but at the cost of column-permutation equivariance violation. Training results indicate that the most flexible architecture is required to prevent from overfitting. Once trained, we apply our best model to the SRSD datasets (Symbolic Regression for Scientific Discovery datasets) which yields state-of-the-art results using the normalized tree-based edit distance, at no extra computational cost.