# Towards Scaling Laws for Symbolic Regression

arXiv: https://arxiv.org/abs/2510.26064 · HTML: https://mlsanity.com/paper/2510.26064/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 18 references that are in the map.
- First posted on arXiv 30 Oct 2025.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Deep Learning for Symbolic Mathematics](https://mlsanity.com/paper/1912.01412/) (2019)
- [End-to-end symbolic regression with transformers](https://mlsanity.com/paper/2204.10532/) (2022)
- [Neural Symbolic Regression that Scales](https://mlsanity.com/paper/2106.06427/) (2021)
- [Small-scale proxies for large-scale Transformer training instabilities](https://mlsanity.com/paper/2309.14322/) (2023)
- [SymbolicGPT: A Generative Transformer Model for Symbolic Regression](https://mlsanity.com/paper/2106.14131/) (2021)
- [Amortized Inference for Causal Structure Learning](https://mlsanity.com/paper/2205.12934/) (2022)
- [Resolving Discrepancies in Compute-Optimal Scaling of Language Models](https://mlsanity.com/paper/2406.19146/) (2024)
- [SymFormer: End-to-end symbolic regression using transformer-based architecture](https://mlsanity.com/paper/2205.15764/) (2022)
- [SNIP: Bridging Mathematical Symbolic and Numeric Realms with Unified Pre-training](https://mlsanity.com/paper/2310.02227/) (2023)
- [Learning in Compact Spaces with Approximately Normalized Transformer](https://mlsanity.com/paper/2505.22014/) (2025)

## Built on by (most cited first, 2 of 2)

- [Shared Symbolic Backbones for Physically Consistent Multi-Output Symbolic Regression](https://mlsanity.com/paper/2607.26528/) (2026)
- [Improving Neural Symbolic Regression with a Diverse Prior and Post-training Alignment](https://mlsanity.com/paper/2610.17994/) (2026)

## Abstract (arXiv)

Symbolic regression (SR) aims to discover the underlying mathematical expressions that explain observed data. This holds promise for both gaining scientific insight and for producing inherently interpretable and generalizable models for tabular data. In this work we focus on the basics of SR. Deep learning-based SR has recently become competitive with genetic programming approaches, but the role of scale has remained largely unexplored. Inspired by scaling laws in language modeling, we present the first systematic investigation of scaling in SR, using a scalable end-to-end transformer pipeline and carefully generated training data. Across five different model sizes and spanning three orders of magnitude in compute, we find that both validation loss and solved rate follow clear power-law trends with compute. We further identify compute-optimal hyperparameter scaling: optimal batch size and learning rate grow with model size, and a token-to-parameter ratio of ≈15 is optimal in our regime, with a slight upward trend as compute increases. These results demonstrate that SR performance is largely predictable from compute and offer important insights for training the next generation of SR models.