# Transformer-based Planning for Symbolic Regression

arXiv: https://arxiv.org/abs/2303.06833 · HTML: https://mlsanity.com/paper/2303.06833/

## Key facts

- Cited by 79 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 51 references that are in the map.
- First posted on arXiv 13 Mar 2023.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [AI Feynman: a Physics-Inspired Method for Symbolic Regression](https://mlsanity.com/paper/1905.11481/) (2019)
- [Discovering Symbolic Models from Deep Learning with Inductive Biases](https://mlsanity.com/paper/2006.11287/) (2020)
- [Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients](https://mlsanity.com/paper/1912.04871/) (2019)
- [Deep Learning for Symbolic Mathematics](https://mlsanity.com/paper/1912.01412/) (2019)
- [Contemporary Symbolic Regression Methods and their Relative Performance](https://mlsanity.com/paper/2107.14351/) (2021)
- [End-to-end symbolic regression with transformers](https://mlsanity.com/paper/2204.10532/) (2022)
- [Neural Symbolic Regression that Scales](https://mlsanity.com/paper/2106.06427/) (2021)
- [Planning with Large Language Models for Code Generation](https://mlsanity.com/paper/2303.05510/) (2023)
- [AI Feynman 2.0: Pareto-optimal symbolic regression exploiting graph modularity](https://mlsanity.com/paper/2006.10782/) (2020)
- [PMLB: A Large Benchmark Suite for Machine Learning Evaluation and Comparison](https://mlsanity.com/paper/1703.00512/) (2017)
- [Sparse identification of nonlinear dynamics for model predictive control in the low-data limit](https://mlsanity.com/paper/1711.05501/) (2017)
- [Sequence-to-Sequence Learning as Beam-Search Optimization](https://mlsanity.com/paper/1606.02960/) (2016)
- [SymbolicGPT: A Generative Transformer Model for Symbolic Regression](https://mlsanity.com/paper/2106.14131/) (2021)
- [Execution-based Code Generation using Deep Reinforcement Learning](https://mlsanity.com/paper/2301.13816/) (2023)
- [Symbolic Regression is NP-hard](https://mlsanity.com/paper/2207.01018/) (2022)
- [Symbolic Physics Learner: Discovering governing equations via Monte Carlo tree search](https://mlsanity.com/paper/2205.13134/) (2022)
- [SymFormer: End-to-end symbolic regression using transformer-based architecture](https://mlsanity.com/paper/2205.15764/) (2022)
- [Machine Translation Decoding beyond Beam Search](https://mlsanity.com/paper/2104.05336/) (2021)
- [Rethinking Symbolic Regression Datasets and Benchmarks for Scientific Discovery](https://mlsanity.com/paper/2206.10540/) (2022)
- [Deep Generative Symbolic Regression with Monte-Carlo-Tree-Search](https://mlsanity.com/paper/2302.11223/) (2023)
- [PPL-MCTS: Constrained Textual Generation Through Discriminator-Guided MCTS Decoding](https://mlsanity.com/paper/2109.13582/) (2021)
- [Evolving symbolic density functionals](https://mlsanity.com/paper/2203.02540/) (2022)
- [OccamNet: A Fast Neural Model for Symbolic Regression at Scale](https://mlsanity.com/paper/2007.10784/) (2020)
- [To Beam Or Not To Beam: That is a Question of Cooperation for Language GANs](https://mlsanity.com/paper/2106.06363/) (2021)

## Built on by (most cited first, 30 of 79)

- [LLM-SR: Scientific Equation Discovery via Programming with Large Language Models](https://mlsanity.com/paper/2404.18400/) (2024)
- [LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models](https://mlsanity.com/paper/2504.10415/) (2025)
- [SNIP: Bridging Mathematical Symbolic and Numeric Realms with Unified Pre-training](https://mlsanity.com/paper/2310.02227/) (2023)
- [Towards Scientific Discovery with Generative AI: Progress, Opportunities, and Challenges](https://mlsanity.com/paper/2412.11427/) (2024)
- [In-Context Symbolic Regression: Leveraging Large Language Models for Function Discovery](https://mlsanity.com/paper/2404.19094/) (2024)
- [Automated Scientific Discovery: From Equation Discovery to Autonomous Discovery Systems](https://mlsanity.com/paper/2305.02251/) (2023)
- [Call for Action: towards the next generation of symbolic regression benchmark](https://mlsanity.com/paper/2505.03977/) (2025)
- [MMSR: Symbolic Regression is a Multi-Modal Information Fusion Task](https://mlsanity.com/paper/2402.18603/) (2024)
- [A Neural Symbolic Model for Space Physics](https://mlsanity.com/paper/2503.07994/) (2025)
- [Priors for symbolic regression](https://mlsanity.com/paper/2304.06333/) (2023)
- [Discovering Mathematical Formulas from Data via GPT-guided Monte Carlo Tree Search](https://mlsanity.com/paper/2401.14424/) (2024)
- [GENSR: Symbolic Regression Based in Equation Generative Space](https://mlsanity.com/paper/2602.20557/) (2026)
- [MDBench: Benchmarking Data-Driven Methods for Model Discovery](https://mlsanity.com/paper/2509.20529/) (2025)
- [LLM-ACES: Closed-Loop Discovery of Dynamical Systems with LLM-Guided Adaptive Search](https://mlsanity.com/paper/2606.25039/) (2026)
- [LLM-Meta-SR: In-Context Learning for Evolving Selection Operators in Symbolic Regression](https://mlsanity.com/paper/2505.18602/) (2025)
- [Improving Monte Carlo Tree Search for Symbolic Regression](https://mlsanity.com/paper/2509.15929/) (2025)
- [ChatSR: Multimodal Large Language Models for Scientific Formula Discovery](https://mlsanity.com/paper/2406.05410/) (2024)
- [Complexity-Aware Deep Symbolic Regression with Robust Risk-Seeking Policy Gradients](https://mlsanity.com/paper/2406.06751/) (2024)
- [Modeling Latent Non-Linear Dynamical System over Time Series](https://mlsanity.com/paper/2412.08114/) (2024)
- [Bridging the Domain Gap in Equation Distillation with Reinforcement Feedback](https://mlsanity.com/paper/2505.15572/) (2025)
- [Diffusion-Based Symbolic Regression](https://mlsanity.com/paper/2505.24776/) (2025)
- [Physics-Grounded Motion Forecasting via Equation Discovery for Trajectory-Guided Image-to-Video Generation](https://mlsanity.com/paper/2507.06830/) (2025)
- [LLM-ODE: Data-driven Discovery of Dynamical Systems with Large Language Models](https://mlsanity.com/paper/2603.20910/) (2026)
- [Generative Pre-Trained Transformer for Symbolic Regression Base In-Context Reinforcement Learning](https://mlsanity.com/paper/2404.06330/) (2024)
- [Active Symbolic Discovery of Ordinary Differential Equations via Phase Portrait Sketching](https://mlsanity.com/paper/2409.01416/) (2024)
- [Ab Initio Nonparametric Variable Selection for Scalable Symbolic Regression with Large p](https://mlsanity.com/paper/2410.13681/) (2024)
- [SatelliteFormula: Multi-Modal Symbolic Regression from Remote Sensing Imagery for Physics Discovery](https://mlsanity.com/paper/2506.06176/) (2025)
- [Analyzing Generalization in Pre-Trained Symbolic Regression](https://mlsanity.com/paper/2509.19849/) (2025)
- [Automated Model Discovery via Multi-modal & Multi-step Pipeline](https://mlsanity.com/paper/2509.25946/) (2025)
- [Data-driven discovery of governing differential equations across physical systems](https://mlsanity.com/paper/2606.09638/) (2026)

## Abstract (arXiv)

Symbolic regression (SR) is a challenging task in machine learning that involves finding a mathematical expression for a function based on its values. Recent advancements in SR have demonstrated the effectiveness of pre-trained transformer-based models in generating equations as sequences, leveraging large-scale pre-training on synthetic datasets and offering notable advantages in terms of inference time over classical Genetic Programming (GP) methods. However, these models primarily rely on supervised pre-training goals borrowed from text generation and overlook equation discovery objectives like accuracy and complexity. To address this, we propose TPSR, a Transformer-based Planning strategy for Symbolic Regression that incorporates Monte Carlo Tree Search into the transformer decoding process. Unlike conventional decoding strategies, TPSR enables the integration of non-differentiable feedback, such as fitting accuracy and complexity, as external sources of knowledge into the transformer-based equation generation process. Extensive experiments on various datasets show that our approach outperforms state-of-the-art methods, enhancing the model's fitting-complexity trade-off, extrapolation abilities, and robustness to noise.