# SymFormer: End-to-end symbolic regression using transformer-based architecture

arXiv: https://arxiv.org/abs/2205.15764 · HTML: https://mlsanity.com/paper/2205.15764/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 48 references that are in the map.
- First posted on arXiv 31 May 2022.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients](https://mlsanity.com/paper/1912.04871/) (2019)
- [Deep Learning for Symbolic Mathematics](https://mlsanity.com/paper/1912.01412/) (2019)
- [End-to-end symbolic regression with transformers](https://mlsanity.com/paper/2204.10532/) (2022)
- [Neural Symbolic Regression that Scales](https://mlsanity.com/paper/2106.06427/) (2021)
- [Learning Equations for Extrapolation and Control](https://mlsanity.com/paper/1806.07259/) (2018)
- [Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks](https://mlsanity.com/paper/1810.00825/) (2018)
- [Extrapolation and learning equations](https://mlsanity.com/paper/1610.02995/) (2016)
- [SymbolicGPT: A Generative Transformer Model for Symbolic Regression](https://mlsanity.com/paper/2106.14131/) (2021)
- [Symbolic Regression via Neural-Guided Genetic Programming Population Seeding](https://mlsanity.com/paper/2111.00053/) (2021)
- [Deep Symbolic Regression for Recurrent Sequences](https://mlsanity.com/paper/2201.04600/) (2022)
- [Interpretable Policies for Reinforcement Learning by Genetic Programming](https://mlsanity.com/paper/1712.04170/) (2017)
- [Analytical Modelling of Exoplanet Transit Specroscopy with Dimensional Analysis and Symbolic Regression](https://mlsanity.com/paper/2112.11600/) (2021)
- [Symbolic Regression Driven by Training Data and Prior Knowledge](https://mlsanity.com/paper/2004.11947/) (2020)
- [Symbolic regression outperforms other models for small data sets](https://mlsanity.com/paper/2103.15147/) (2021)
- [Symbolic Regression Methods for Reinforcement Learning](https://mlsanity.com/paper/1903.09688/) (2019)
- [Informed Equation Learning](https://mlsanity.com/paper/2105.06331/) (2021)
- [Augmenting astrophysical scaling relations with machine learning: application to reducing the Sunyaev-Zeldovich flux-mass scatter](https://mlsanity.com/paper/2201.01305/) (2022)

## Built on by (most cited first, 30 of 59)

- [Transformer-based Planning for Symbolic Regression](https://mlsanity.com/paper/2303.06833/) (2023)
- [Symbolic Regression is NP-hard](https://mlsanity.com/paper/2207.01018/) (2022)
- [ODEFormer: Symbolic Regression of Dynamical Systems with Transformers](https://mlsanity.com/paper/2310.05573/) (2023)
- [Predicting Ordinary Differential Equations with Transformers](https://mlsanity.com/paper/2307.12617/) (2023)
- [Symbol: Generating Flexible Black-Box Optimizers through Symbolic Equation Learning](https://mlsanity.com/paper/2402.02355/) (2024)
- [DrSR: LLM based Scientific Equation Discovery with Dual Reasoning from Data and Experience](https://mlsanity.com/paper/2506.04282/) (2025)
- [Controllable Neural Symbolic Regression](https://mlsanity.com/paper/2304.10336/) (2023)
- [Toward Physically Plausible Data-Driven Models: A Novel Neural Network Approach to Symbolic Regression](https://mlsanity.com/paper/2302.00773/) (2023)
- [LLM4ED: Large Language Models for Automatic Equation Discovery](https://mlsanity.com/paper/2405.07761/) (2024)
- [MMSR: Symbolic Regression is a Multi-Modal Information Fusion Task](https://mlsanity.com/paper/2402.18603/) (2024)
- [Priors for symbolic regression](https://mlsanity.com/paper/2304.06333/) (2023)
- [SymbolNet: Neural Symbolic Regression with Adaptive Dynamic Pruning for Compression](https://mlsanity.com/paper/2401.09949/) (2024)
- [A Comparison of Recent Algorithms for Symbolic Regression to Genetic Programming](https://mlsanity.com/paper/2406.03585/) (2024)
- [ChatSR: Multimodal Large Language Models for Scientific Formula Discovery](https://mlsanity.com/paper/2406.05410/) (2024)
- [Al-Khwarizmi: Discovering Physical Laws with Foundation Models](https://mlsanity.com/paper/2502.01702/) (2025)
- [LLM-Based Scientific Equation Discovery via Physics-Informed Token-Regularized Policy Optimization](https://mlsanity.com/paper/2602.10576/) (2026)
- [Differentiable Genetic Programming for High-dimensional Symbolic Regression](https://mlsanity.com/paper/2304.08915/) (2023)
- [Complexity-Aware Deep Symbolic Regression with Robust Risk-Seeking Policy Gradients](https://mlsanity.com/paper/2406.06751/) (2024)
- [Learning Interpretable Network Dynamics via Universal Neural Symbolic Regression](https://mlsanity.com/paper/2411.06833/) (2024)
- [Diffusion-Based Symbolic Regression](https://mlsanity.com/paper/2505.24776/) (2025)
- [Discovering ordinary differential equations that govern time-series](https://mlsanity.com/paper/2211.02830/) (2022)
- [Generative Pre-Trained Transformer for Symbolic Regression Base In-Context Reinforcement Learning](https://mlsanity.com/paper/2404.06330/) (2024)
- [SatelliteFormula: Multi-Modal Symbolic Regression from Remote Sensing Imagery for Physics Discovery](https://mlsanity.com/paper/2506.06176/) (2025)
- [Analyzing Generalization in Pre-Trained Symbolic Regression](https://mlsanity.com/paper/2509.19849/) (2025)
- [Data-driven discovery of governing differential equations across physical systems](https://mlsanity.com/paper/2606.09638/) (2026)
- [Generalizing the SINDy approach with nested neural networks](https://mlsanity.com/paper/2404.15742/) (2024)
- [Interchangeable Token Embeddings for Extendable Vocabulary and Alpha-Equivalence](https://mlsanity.com/paper/2410.17161/) (2024)
- [ViSymRe: Vision Multimodal Symbolic Regression](https://mlsanity.com/paper/2412.11139/) (2024)
- [Sparse Interpretable Deep Learning with LIES Networks for Symbolic Regression](https://mlsanity.com/paper/2506.08267/) (2025)
- [Data-driven Discovery of Digital Twins in Biomedical Research](https://mlsanity.com/paper/2508.21484/) (2025)

## Abstract (arXiv)

Many real-world problems can be naturally described by mathematical formulas. The task of finding formulas from a set of observed inputs and outputs is called symbolic regression. Recently, neural networks have been applied to symbolic regression, among which the transformer-based ones seem to be the most promising. After training the transformer on a large number of formulas (in the order of days), the actual inference, i.e., finding a formula for new, unseen data, is very fast (in the order of seconds). This is considerably faster than state-of-the-art evolutionary methods. The main drawback of transformers is that they generate formulas without numerical constants, which have to be optimized separately, so yielding suboptimal results. We propose a transformer-based approach called SymFormer, which predicts the formula by outputting the individual symbols and the corresponding constants simultaneously. This leads to better performance in terms of fitting the available data. In addition, the constants provided by SymFormer serve as a good starting point for subsequent tuning via gradient descent to further improve the performance. We show on a set of benchmarks that SymFormer outperforms two state-of-the-art methods while having faster inference.