mlsanityData to 14 Oct 2026

End-to-end symbolic regression with transformers

arXiv 2204.10532 ↗First posted 22 Apr 2022cs.LGcited by 203

Not on the map. Why?

Abstract (arXiv)

Symbolic regression, the task of predicting the mathematical expression of a function from the observation of its values, is a difficult task which usually involves a two-step procedure: predicting the "skeleton" of the expression up to the choice of numerical constants, then fitting the constants by optimizing a non-convex loss function. The dominant approach is genetic programming, which evolves candidates by iterating this subroutine a large number of times. Neural networks have recently been tasked to predict the correct skeleton in a single try, but remain much less powerful. In this paper, we challenge this two-step procedure, and task a Transformer to directly predict the full mathematical expression, constants included. One can subsequently refine the predicted constants by feeding them to the non-convex optimizer as an informed initialization. We present ablations to show that this end-to-end approach yields better results, sometimes even without the refinement step. We evaluate our model on problems from the SRBench benchmark and show that our model approaches the performance of state-of-the-art genetic programming with several orders of magnitude faster inference.

Key facts. Cited by 203 papers in the map (data through 14 Oct 2026). Builds on 24 of its 44 references that are in the map. First posted on arXiv 22 Apr 2022.

Builds on

24 of 44 references.

References · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019AI Feynman: a Physics-Inspired Method for Symbolic Regressioncited by 548
  3. 2020Generative Language Modeling for Automated Theorem Provingcited by 388
  4. 2020Discovering Symbolic Models from Deep Learning with Inductive Biasescited by 344
  5. 2019Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradientscited by 309
  6. 2019Deep Learning for Symbolic Mathematicscited by 270
  7. 2021Contemporary Symbolic Regression Methods and their Relative Performancecited by 215
  8. 2021Neural Symbolic Regression that Scalescited by 192
  9. 2018Learning Equations for Extrapolation and Controlcited by 156
  10. 2016Sequence-to-Sequence Learning as Beam-Search Optimizationcited by 127
  11. 2016Extrapolation and learning equationscited by 110
  12. 2021SymbolicGPT: A Generative Transformer Model for Symbolic Regressioncited by 105
12 more
  1. 2021Symbolic Regression via Neural-Guided Genetic Programming Population Seedingcited by 101
  2. 2019Bayesian Symbolic Regressioncited by 78
  3. 2021Linear algebra with transformerscited by 63
  4. 2020A Bayesian machine scientist to aid in the solution of challenging scientific problemscited by 61
  5. 2022Deep Symbolic Regression for Recurrent Sequencescited by 54
  6. 2020Teaching Temporal Logics to Neural Networkscited by 51
  7. 2016Towards Deep Symbolic Reinforcement Learningcited by 42
  8. 2018Learning concise representations for regression by evolving networks of treescited by 33
  9. 2021Symbolic Brittleness in Sequence Models: on Systematic Generalization in Symbolic Mathematicscited by 27
  10. 2020Symbolic Pregression: Discovering Physical Laws from Distorted Videocited by 21
  11. 2019Symbolic Regression Methods for Reinforcement Learningcited by 11
  12. 2021Accelerating Understanding of Scientific Experiments with End to End Symbolic Regressioncited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 13; 2023: 32; 2024: 48; 2025: 63; 2026: 47.

203 papers.

Sort:

By area:

  1. 2024LLM-SR: Scientific Equation Discovery via Programming with Large Language Modelscited by 133
  2. 2022Interpretable Scientific Discovery with Symbolic Regression: A Reviewcited by 105
  3. 2025From Automation to Autonomy: A Survey on Large Language Models in Scientific Discoverycited by 93
  4. 2023Transformer-based Planning for Symbolic Regressioncited by 79
  5. 2022Symbolic Regression is NP-hardcited by 71
  6. 2025LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Modelscited by 71
  7. 2023ODEFormer: Symbolic Regression of Dynamical Systems with Transformerscited by 62
  8. 2022SymFormer: End-to-end symbolic regression using transformer-based architecturecited by 59
  9. 2023Interpretable Machine Learning for Science with PySR and SymbolicRegression.jlcited by 54
  10. 2023SNIP: Bridging Mathematical Symbolic and Numeric Realms with Unified Pre-trainingcited by 47
  11. 2022Rethinking Symbolic Regression Datasets and Benchmarks for Scientific Discoverycited by 46
  12. 2024Towards a Foundation Model for Partial Differential Equations: Multi-Operator Learning and Extrapolationcited by 46
38 more
  1. 2023Deep Generative Symbolic Regression with Monte-Carlo-Tree-Searchcited by 41
  2. 2024Towards Scientific Discovery with Generative AI: Progress, Opportunities, and Challengescited by 41
  3. 2025NewtonBench: Benchmarking Generalizable Scientific Law Discovery in LLM Agentscited by 36
  4. 2025SR-Scientist: Scientific Equation Discovery With Agentic AIcited by 35
  5. 2024PROSE-FD: A Multimodal PDE Foundation Model for Learning Multiple Operators for Forecasting Fluid Dynamicscited by 34
  6. 2024In-Context Symbolic Regression: Leveraging Large Language Models for Function Discoverycited by 31
  7. 2023Deep Generative Symbolic Regressioncited by 28
  8. 2023Predicting Ordinary Differential Equations with Transformerscited by 26
  9. 2023Deep Learning Symmetries and Their Lie Groups, Algebras, and Subalgebras from First Principlescited by 22
  10. 2025DrSR: LLM based Scientific Equation Discovery with Dual Reasoning from Data and Experiencecited by 22
  11. 2023Controllable Neural Symbolic Regressioncited by 20
  12. 2025Position: The Future of Bayesian Prediction Is Prior-Fittedcited by 20
  13. 2023Automated Scientific Discovery: From Equation Discovery to Autonomous Discovery Systemscited by 19
  14. 2025A Comprehensive Survey on Self-Interpretable Neural Networkscited by 18
  15. 2024Symbolic regression via MDLformer-guided search: from minimizing prediction error to minimizing description lengthcited by 17
  16. 2025Call for Action: towards the next generation of symbolic regression benchmarkcited by 17
  17. 2023A Neural-Guided Dynamic Symbolic Network for Exploring Mathematical Expressions from Datacited by 16
  18. 2023PROSE: Predicting Operators and Symbolic Expressions using Multimodal Transformerscited by 16
  19. 2024LeMON: Learning to Learn Multi-Operator Networkscited by 14
  20. 2025Interpretable Machine Learning in Physics: A Reviewcited by 14
  21. 2023Efficient Generator of Mathematical Expressions for Symbolic Regressioncited by 13
  22. 2024MMSR: Symbolic Regression is a Multi-Modal Information Fusion Taskcited by 13
  23. 2024The Causal Chambers: Real Physical Systems as a Testbed for AI Methodologycited by 12
  24. 2024Meta-Designing Quantum Experiments with Language Modelscited by 12
  25. 2026A Probabilistic Framework for LLM-Based Model Discoverycited by 12
  26. 2022GSR: A Generalized Symbolic Regression Approachcited by 11
  27. 2025A Neural Symbolic Model for Space Physicscited by 11
  28. 2023Priors for symbolic regressioncited by 10
  29. 2023ParFam -- (Neural Guided) Symbolic Regression Based on Continuous Global Optimizationcited by 10
  30. 2023A Transformer Model for Symbolic Regression towards Scientific Discoverycited by 9
  31. 2024Self-supervised contrastive learning performs non-linear system identificationcited by 9
  32. 2026GENSR: Symbolic Regression Based in Equation Generative Spacecited by 9
  33. 2022Symbolic Recovery of Differential Equations: The Identifiability Problemcited by 8
  34. 2022Is the Machine Smarter than the Theorist: Deriving Formulas for Particle Kinematics with Symbolic Regressioncited by 8
  35. 2023RSRM: Reinforcement Symbolic Regression Machinecited by 8
  36. 2022Symbolic Expression Transformer: A Computer Vision Approach for Symbolic Regressioncited by 7
  37. 2023Learning to Compute Gr"obner Basescited by 7
  38. 2023Explainable Fraud Detection with Deep Symbolic Classificationcited by 7

50 most cited of 203 All 203 (JSON)