# Analyzing Generalization in Pre-Trained Symbolic Regression

arXiv: https://arxiv.org/abs/2509.19849 · HTML: https://mlsanity.com/paper/2509.19849/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 36 references that are in the map.
- First posted on arXiv 24 Sep 2025.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [AI Feynman: a Physics-Inspired Method for Symbolic Regression](https://mlsanity.com/paper/1905.11481/) (2019)
- [Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients](https://mlsanity.com/paper/1912.04871/) (2019)
- [Deep Learning for Symbolic Mathematics](https://mlsanity.com/paper/1912.01412/) (2019)
- [Contemporary Symbolic Regression Methods and their Relative Performance](https://mlsanity.com/paper/2107.14351/) (2021)
- [End-to-end symbolic regression with transformers](https://mlsanity.com/paper/2204.10532/) (2022)
- [Neural Symbolic Regression that Scales](https://mlsanity.com/paper/2106.06427/) (2021)
- [LLM-SR: Scientific Equation Discovery via Programming with Large Language Models](https://mlsanity.com/paper/2404.18400/) (2024)
- [Symbolic Regression with a Learned Concept Library](https://mlsanity.com/paper/2409.09359/) (2024)
- [Transformer-based Planning for Symbolic Regression](https://mlsanity.com/paper/2303.06833/) (2023)
- [Symbolic Regression is NP-hard](https://mlsanity.com/paper/2207.01018/) (2022)
- [LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models](https://mlsanity.com/paper/2504.10415/) (2025)
- [SymFormer: End-to-end symbolic regression using transformer-based architecture](https://mlsanity.com/paper/2205.15764/) (2022)
- [Rethinking Symbolic Regression Datasets and Benchmarks for Scientific Discovery](https://mlsanity.com/paper/2206.10540/) (2022)
- [Deep Generative Symbolic Regression with Monte-Carlo-Tree-Search](https://mlsanity.com/paper/2302.11223/) (2023)
- [Discovering Causal Relations and Equations from Data](https://mlsanity.com/paper/2305.13341/) (2023)
- [In-Context Symbolic Regression: Leveraging Large Language Models for Function Discovery](https://mlsanity.com/paper/2404.19094/) (2024)
- [Discovering a reaction-diffusion model for Alzheimer's disease by combining PINNs with symbolic regression](https://mlsanity.com/paper/2307.08107/) (2023)
- [Symbolic Pregression: Discovering Physical Laws from Distorted Video](https://mlsanity.com/paper/2005.11212/) (2020)
- [Automatic generation of interpretable hyperelastic material models by symbolic regression](https://mlsanity.com/paper/2206.15099/) (2022)
- [How to decay your learning rate](https://mlsanity.com/paper/2103.12682/) (2021)
- [Discovering Mathematical Formulas from Data via GPT-guided Monte Carlo Tree Search](https://mlsanity.com/paper/2401.14424/) (2024)
- [Symbolic Regression on FPGAs for Fast Machine Learning Inference](https://mlsanity.com/paper/2305.04099/) (2023)
- [A Transformer Model for Symbolic Regression towards Scientific Discovery](https://mlsanity.com/paper/2312.04070/) (2023)
- [Scaling Up Unbiased Search-based Symbolic Regression](https://mlsanity.com/paper/2506.19626/) (2025)

## Built on by (most cited first, 3 of 3)

- [Multi-Modal Learning meets Genetic Programming: Analyzing Alignment in Latent Space Optimization](https://mlsanity.com/paper/2604.08324/) (2026)
- [SNIP++: Fine-Grained Symbolic-Numerical Alignment for Symbolic Regression](https://mlsanity.com/paper/2609.31965/) (2026)
- [Synthesizing Physics Formulae with Transformers](https://mlsanity.com/paper/2610.03947/) (2026)

## Abstract (arXiv)

Symbolic regression algorithms search a space of mathematical expressions for formulas that explain given data. Transformer-based models have emerged as a promising, scalable approach shifting the expensive combinatorial search to a large-scale pre-training phase. However, the success of these models is critically dependent on their pre-training data. Their ability to generalize to problems outside of this pre-training distribution remains largely unexplored. In this work, we conduct a systematic empirical study to evaluate the generalization capabilities of pre-trained, transformer-based symbolic regression. We rigorously test performance both within the pre-training distribution and on a series of out-of-distribution challenges for several state of the art approaches. Our findings reveal a significant dichotomy: while pre-trained models perform well in-distribution, the performance consistently degrades in out-of-distribution scenarios. We conclude that this generalization gap is a critical barrier for practitioners, as it severely limits the practical use of pre-trained approaches for real-world applications.