# Symbolic regression outperforms other models for small data sets

arXiv: https://arxiv.org/abs/2103.15147 · HTML: https://mlsanity.com/paper/2103.15147/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 26 references that are in the map.
- First posted on arXiv 28 Mar 2021.

## Builds on (most cited first, 9 of 9)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [AI Feynman: a Physics-Inspired Method for Symbolic Regression](https://mlsanity.com/paper/1905.11481/) (2019)
- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [Discovering Symbolic Models from Deep Learning with Inductive Biases](https://mlsanity.com/paper/2006.11287/) (2020)
- [Integration of Neural Network-Based Symbolic Regression in Deep Learning for Scientific Discovery](https://mlsanity.com/paper/1912.04825/) (2019)
- [AI Poincar'e: Machine Learning Conservation Laws from Trajectories](https://mlsanity.com/paper/2011.04698/) (2020)
- [PMLB v1.0: An open source dataset collection for benchmarking machine learning methods](https://mlsanity.com/paper/2012.00058/) (2020)
- [Learning Symbolic Physics with Graph Networks](https://mlsanity.com/paper/1909.05862/) (2019)

## Built on by (most cited first, 12 of 12)

- [SymFormer: End-to-end symbolic regression using transformer-based architecture](https://mlsanity.com/paper/2205.15764/) (2022)
- [AI-Aristotle: A Physics-Informed framework for Systems Biology Gray-Box Identification](https://mlsanity.com/paper/2310.01433/) (2023)
- [Toward Physically Plausible Data-Driven Models: A Novel Neural Network Approach to Symbolic Regression](https://mlsanity.com/paper/2302.00773/) (2023)
- [Augmenting astrophysical scaling relations with machine learning: application to reducing the Sunyaev-Zeldovich flux-mass scatter](https://mlsanity.com/paper/2201.01305/) (2022)
- [LLM-ODE: Data-driven Discovery of Dynamical Systems with Large Language Models](https://mlsanity.com/paper/2603.20910/) (2026)
- [What can we Learn by Predicting Accuracy?](https://mlsanity.com/paper/2208.01358/) (2022)
- [Hyperdimensional computing: a fast, robust and interpretable paradigm for biological data](https://mlsanity.com/paper/2402.17572/) (2024)
- [Generalizing the SINDy approach with nested neural networks](https://mlsanity.com/paper/2404.15742/) (2024)
- [An Efficient and Generalizable Symbolic Regression Method for Time Series Analysis](https://mlsanity.com/paper/2409.03986/) (2024)
- [Information Fusion via Symbolic Regression: A Tutorial in the Context of Human Health](https://mlsanity.com/paper/2306.00153/) (2023)
- [Neuro-Evolutionary Approach to Physics-Aware Symbolic Regression](https://mlsanity.com/paper/2504.16503/) (2025)
- [Turning Time Series into Algebraic Equations: Symbolic Machine Learning for Interpretable Modeling of Chaotic Time Series](https://mlsanity.com/paper/2603.07261/) (2026)

## Abstract (arXiv)

Machine learning is often applied in health science to obtain predictions and new understandings of complex phenomena and relationships, but an availability of sufficient data for model training is a widespread problem. Traditional machine learning techniques, such as random forests and gradient boosting, tend to overfit when working with data sets of only a few hundred observations. This study demonstrates that for small training sets of 250 observations, symbolic regression generalises better to out-of-sample data than traditional machine learning frameworks, as measured by the coefficient of determination R2 on the validation set. In 132 out of 240 cases, symbolic regression achieves a higher R2 than any of the other models on the out-of-sample data. Furthermore, symbolic regression also preserves the interpretability of linear models and decision trees, an added benefit to its superior generalisation. The second best algorithm was found to be a random forest, which performs best in 37 of the 240 cases. When restricting the comparison to interpretable models, symbolic regression performs best in 184 out of 240 cases.