mlsanityData to 14 Oct 2026

Neural Symbolic Regression that Scales

arXiv 2106.06427 ↗First posted 11 Jun 2021cs.LGcited by 192

Not on the map. Why?

Abstract (arXiv)

Symbolic equations are at the core of scientific discovery. The task of discovering the underlying equation from a set of input-output pairs is called symbolic regression. Traditionally, symbolic regression methods use hand-designed strategies that do not improve with experience. In this paper, we introduce the first symbolic regression method that leverages large scale pre-training. We procedurally generate an unbounded set of equations, and simultaneously pre-train a Transformer to predict the symbolic equation from a corresponding set of input-output-pairs. At test time, we query the model on a new set of points and use its output to guide the search for the equation. We show empirically that this approach can re-discover a set of well-known physical equations, and that it improves over time with more data and compute.

Key facts. Cited by 192 papers in the map (data through 14 Oct 2026). Builds on 21 of its 40 references that are in the map. First posted on arXiv 11 Jun 2021.

Builds on

21 of 40 references.

References · 21

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  9. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  10. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  11. 2019AI Feynman: a Physics-Inspired Method for Symbolic Regressioncited by 548
  12. 2020Generative Language Modeling for Automated Theorem Provingcited by 388
9 more
  1. 2016The CMA Evolution Strategy: A Tutorialcited by 377
  2. 2020Discovering Symbolic Models from Deep Learning with Inductive Biasescited by 344
  3. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 335
  4. 2019Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradientscited by 309
  5. 2017Grammar Variational Autoencodercited by 285
  6. 2019Deep Learning for Symbolic Mathematicscited by 270
  7. 2018Learning Equations for Extrapolation and Controlcited by 156
  8. 2020AI Feynman 2.0: Pareto-optimal symbolic regression exploiting graph modularitycited by 144
  9. 2016Extrapolation and learning equationscited by 110

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 23; 2023: 32; 2024: 40; 2025: 46; 2026: 49.

192 papers.

Sort:

By area:

  1. 2022End-to-end symbolic regression with transformerscited by 203
  2. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 141
  3. 2024LLM-SR: Scientific Equation Discovery via Programming with Large Language Modelscited by 133
  4. 2021SymbolicGPT: A Generative Transformer Model for Symbolic Regressioncited by 105
  5. 2022Interpretable Scientific Discovery with Symbolic Regression: A Reviewcited by 105
  6. 2024LLM and Simulation as Bilevel Optimizers: A New Paradigm to Advance Physical Scientific Discoverycited by 86
  7. 2023Transformer-based Planning for Symbolic Regressioncited by 79
  8. 2025LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Modelscited by 71
  9. 2021Linear algebra with transformerscited by 63
  10. 2022SymFormer: End-to-end symbolic regression using transformer-based architecturecited by 59
  11. 2022Deep Symbolic Regression for Recurrent Sequencescited by 54
  12. 2023SNIP: Bridging Mathematical Symbolic and Numeric Realms with Unified Pre-trainingcited by 47
38 more
  1. 2022Rethinking Symbolic Regression Datasets and Benchmarks for Scientific Discoverycited by 46
  2. 2024Discovering Preference Optimization Algorithms with and for Large Language Modelscited by 42
  3. 2023Deep Generative Symbolic Regression with Monte-Carlo-Tree-Searchcited by 41
  4. 2024Towards Scientific Discovery with Generative AI: Progress, Opportunities, and Challengescited by 41
  5. 2023Discovering Causal Relations and Equations from Datacited by 36
  6. 2022Towards Data-and Knowledge-Driven Artificial Intelligence: A Survey on Neuro-Symbolic Computingcited by 35
  7. 2025SR-Scientist: Scientific Equation Discovery With Agentic AIcited by 35
  8. 2024In-Context Symbolic Regression: Leveraging Large Language Models for Function Discoverycited by 31
  9. 2024Global Lyapunov functions: a long-standing open problem in mathematics, with symbolic transformerscited by 29
  10. 2022SALSA: Attacking Lattice Cryptography with Transformerscited by 28
  11. 2023Deep Generative Symbolic Regressioncited by 28
  12. 2024AlphaForge: A Framework to Mine and Dynamically Combine Formulaic Alpha Factorscited by 27
  13. 2023Predicting Ordinary Differential Equations with Transformerscited by 26
  14. 2024Symbol: Generating Flexible Black-Box Optimizers through Symbolic Equation Learningcited by 22
  15. 2025DrSR: LLM based Scientific Equation Discovery with Dual Reasoning from Data and Experiencecited by 22
  16. 2023Controllable Neural Symbolic Regressioncited by 20
  17. 2022Taylor Genetic Programming for Symbolic Regressioncited by 19
  18. 2022What is my math transformer doing? -- Three results on interpretability and generalizationcited by 19
  19. 2023Automated Scientific Discovery: From Equation Discovery to Autonomous Discovery Systemscited by 19
  20. 2025A Comprehensive Survey on Self-Interpretable Neural Networkscited by 18
  21. 2024Symbolic regression via MDLformer-guided search: from minimizing prediction error to minimizing description lengthcited by 17
  22. 2025Call for Action: towards the next generation of symbolic regression benchmarkcited by 17
  23. 2023A Neural-Guided Dynamic Symbolic Network for Exploring Mathematical Expressions from Datacited by 16
  24. 2026Think like a Scientist: Physics-guided LLM Agent for Equation Discoverycited by 16
  25. 2023Toward Physically Plausible Data-Driven Models: A Novel Neural Network Approach to Symbolic Regressioncited by 14
  26. 2024LLM4ED: Large Language Models for Automatic Equation Discoverycited by 14
  27. 2025Graph ODEs and Beyond: A Comprehensive Survey on Integrating Differential Equations with Graph Neural Networkscited by 14
  28. 2023Efficient Generator of Mathematical Expressions for Symbolic Regressioncited by 13
  29. 2024MMSR: Symbolic Regression is a Multi-Modal Information Fusion Taskcited by 13
  30. 2022GSR: A Generalized Symbolic Regression Approachcited by 11
  31. 2025A Neural Symbolic Model for Space Physicscited by 11
  32. 2023Priors for symbolic regressioncited by 10
  33. 2023ParFam -- (Neural Guided) Symbolic Regression Based on Continuous Global Optimizationcited by 10
  34. 2024Discovering Mathematical Formulas from Data via GPT-guided Monte Carlo Tree Searchcited by 10
  35. 2024Neuro-Symbolic AI: Explainability, Challenges, and Future Trendscited by 10
  36. 2023A Transformer Model for Symbolic Regression towards Scientific Discoverycited by 9
  37. 2026GENSR: Symbolic Regression Based in Equation Generative Spacecited by 9
  38. 2022Symbolic Recovery of Differential Equations: The Identifiability Problemcited by 8

50 most cited of 192 All 192 (JSON)