# In Context Learning and Reasoning for Symbolic Regression with Large Language Models

arXiv: https://arxiv.org/abs/2410.17448 · HTML: https://mlsanity.com/paper/2410.17448/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Mathematical reasoning, one of 17 directions in LLM reasoning.
- Builds on 36 of its 103 references that are in the map.
- First posted on arXiv 22 Oct 2024.

## Placed in

- LLM reasoning / [Mathematical reasoning](https://mlsanity.com/lr/mathematical-reasoning/)

## Builds on (most cited first, 30 of 36)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Show Your Work: Scratchpads for Intermediate Computation with Language Models](https://mlsanity.com/paper/2112.00114/) (2021)
- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [Learning How to Ask: Querying LMs with Mixtures of Soft Prompts](https://mlsanity.com/paper/2104.06599/) (2021)
- [Beyond Goldfish Memory: Long-Term Open-Domain Conversation](https://mlsanity.com/paper/2107.07567/) (2021)
- [Music Transformer](https://mlsanity.com/paper/1809.04281/) (2018)
- [Neural Legal Judgment Prediction in English](https://mlsanity.com/paper/1906.02059/) (2019)
- [PaperQA: Retrieval-Augmented Generative Agent for Scientific Research](https://mlsanity.com/paper/2312.07559/) (2023)
- [LLM-SR: Scientific Equation Discovery via Programming with Large Language Models](https://mlsanity.com/paper/2404.18400/) (2024)
- [Language Model Crossover: Variation through Few-Shot Prompting](https://mlsanity.com/paper/2302.12170/) (2023)
- [A Survey on Large Language Model-Based Game Agents](https://mlsanity.com/paper/2404.02039/) (2024)
- [Long Time No See! Open-Domain Conversation with Long-Term Persona Memory](https://mlsanity.com/paper/2203.05797/) (2022)
- [SymbolicGPT: A Generative Transformer Model for Symbolic Regression](https://mlsanity.com/paper/2106.14131/) (2021)
- [A Review of Large Language Models and Autonomous Agents in Chemistry](https://mlsanity.com/paper/2407.01603/) (2024)
- [Symbolic Regression with a Learned Concept Library](https://mlsanity.com/paper/2409.09359/) (2024)
- [Bayesian Symbolic Regression](https://mlsanity.com/paper/1910.08892/) (2019)
- [Keep Me Updated! Memory Management in Long-term Conversations](https://mlsanity.com/paper/2210.08750/) (2022)
- [Deep symbolic regression for physics guided by units constraints: toward the automated discovery of physical laws](https://mlsanity.com/paper/2303.03192/) (2023)
- [Deep Symbolic Regression for Recurrent Sequences](https://mlsanity.com/paper/2201.04600/) (2022)
- [Interpretable Machine Learning for Science with PySR and SymbolicRegression.jl](https://mlsanity.com/paper/2305.01582/) (2023)
- [Less is More: Learning to Refine Dialogue History for Personalized Dialogue Generation](https://mlsanity.com/paper/2204.08128/) (2022)
- [Evolving Code with A Large Language Model](https://mlsanity.com/paper/2401.07102/) (2024)
- [Shape-constrained Symbolic Regression -- Improving Extrapolation with Prior Knowledge](https://mlsanity.com/paper/2103.15624/) (2021)
- [Can LLMs Master Math? Investigating Large Language Models on Math Stack Exchange](https://mlsanity.com/paper/2404.00344/) (2024)
- [Pre-trained Language Models as Prior Knowledge for Playing Text-based Games](https://mlsanity.com/paper/2107.08408/) (2021)

## Built on by (most cited first, 7 of 7)

- [Think like a Scientist: Physics-guided LLM Agent for Equation Discovery](https://mlsanity.com/paper/2602.12259/) (2026)
- [Knowledge Integration for Physics-informed Symbolic Regression Using Pre-trained Large Language Models](https://mlsanity.com/paper/2509.03036/) (2025)
- [Large Language Models for Extrapolative Modeling of Manufacturing Processes](https://mlsanity.com/paper/2502.12185/) (2025)
- [Automated Model Discovery via Multi-modal & Multi-step Pipeline](https://mlsanity.com/paper/2509.25946/) (2025)
- [Data-driven Discovery of Digital Twins in Biomedical Research](https://mlsanity.com/paper/2508.21484/) (2025)
- [Can Test-time Computation Mitigate Reproduction Bias in Neural Symbolic Regression?](https://mlsanity.com/paper/2505.22081/) (2025)
- [LLM-driven design of physics-constrained constitutive models: two agents are better than one](https://mlsanity.com/paper/2605.23754/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are transformer-based machine learning models that have shown remarkable performance in tasks for which they were not explicitly trained. Here, we explore the potential of LLMs to perform symbolic regression -- a machine-learning method for finding simple and accurate equations from datasets. We prompt GPT-4 and GPT-4o models to suggest expressions from data, which are then optimized and evaluated using external Python tools. These results are fed back to the LLMs, which propose improved expressions while optimizing for complexity and loss. Using chain-of-thought prompting, we instruct the models to analyze data, prior expressions, and the scientific context (expressed in natural language) for each problem before generating new expressions. We evaluated the workflow in rediscovery of Langmuir and dual-site Langmuir's model for adsorption, along with Nikuradse's dataset on flow in rough pipes, which does not have a known target model equation. Both the GPT-4 and GPT-4o models successfully rediscovered equations, with better performance when using a scratchpad and considering scientific context. GPT-4o model demonstrated improved reasoning with data patterns, particularly evident in the dual-site Langmuir and Nikuradse dataset. We demonstrate how strategic prompting improves the model's performance and how the natural language interface simplifies integrating theory with data. We also applied symbolic mathematical constraints based on the background knowledge of data via prompts and found that LLMs generate meaningful equations more frequently. Although this approach does not outperform established SR programs where target equations are more complex, LLMs can nonetheless iterate toward improved solutions while following instructions and incorporating scientific context in natural language.