# Symbolic Regression as Feature Engineering Method for Machine and Deep Learning Regression Tasks

arXiv: https://arxiv.org/abs/2311.06028 · HTML: https://mlsanity.com/paper/2311.06028/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 95 references that are in the map.
- First posted on arXiv 10 Nov 2023.

## Builds on (most cited first, 22 of 22)

- [AI Feynman: a Physics-Inspired Method for Symbolic Regression](https://mlsanity.com/paper/1905.11481/) (2019)
- [AutoML: A Survey of the State-of-the-Art](https://mlsanity.com/paper/1908.00709/) (2019)
- [Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients](https://mlsanity.com/paper/1912.04871/) (2019)
- [A Survey on the Explainability of Supervised Machine Learning](https://mlsanity.com/paper/2011.07876/) (2020)
- [Contemporary Symbolic Regression Methods and their Relative Performance](https://mlsanity.com/paper/2107.14351/) (2021)
- [Hidden Physics Models: Machine Learning of Nonlinear Partial Differential Equations](https://mlsanity.com/paper/1708.00588/) (2017)
- [Auto-Keras: An Efficient Neural Architecture Search System](https://mlsanity.com/paper/1806.10282/) (2018)
- [Sparse identification of nonlinear dynamics for model predictive control in the low-data limit](https://mlsanity.com/paper/1711.05501/) (2017)
- [PySINDy: A comprehensive Python package for robust sparse system identification](https://mlsanity.com/paper/2111.08481/) (2021)
- [Interpretability and Explainability: A Machine Learning Zoo Mini-tour](https://mlsanity.com/paper/2012.01805/) (2020)
- [Forecast Evaluation for Data Scientists: Common Pitfalls and Best Practices](https://mlsanity.com/paper/2203.10716/) (2022)
- [Where are we now? A large benchmark study of recent symbolic regression methods](https://mlsanity.com/paper/1804.09331/) (2018)
- [A computational framework for physics-informed symbolic regression with straightforward integration of domain knowledge](https://mlsanity.com/paper/2209.06257/) (2022)
- [AutoField: Automating Feature Selection in Deep Recommender Systems](https://mlsanity.com/paper/2204.09078/) (2022)
- [An Empirical Analysis of Feature Engineering for Predictive Modeling](https://mlsanity.com/paper/1701.07852/) (2017)
- [Shape-constrained Symbolic Regression -- Improving Extrapolation with Prior Knowledge](https://mlsanity.com/paper/2103.15624/) (2021)
- [Survey on Feature Selection](https://mlsanity.com/paper/1510.02892/) (2015)
- [Relational Autoencoder for Feature Extraction](https://mlsanity.com/paper/1802.03145/) (2018)
- [Ease.ml: Towards Multi-tenant Resource Sharing for Machine Learning Workloads](https://mlsanity.com/paper/1708.07308/) (2017)
- [SubStrat: A Subset-Based Strategy for Faster AutoML](https://mlsanity.com/paper/2206.03070/) (2022)
- [A Probabilistic Linear Genetic Programming with Stochastic Context-Free Grammar for solving Symbolic Regression problems](https://mlsanity.com/paper/1704.00828/) (2017)
- [Accelerating Genetic Programming using GPUs](https://mlsanity.com/paper/2110.11226/) (2021)

## Built on by (most cited first, 5 of 5)

- [Introducing 'Inside' Out of Distribution](https://mlsanity.com/paper/2407.04534/) (2024)
- [Pulling the Carpet Below the Learner's Feet: Genetic Algorithm To Learn Ensemble Machine Learning Model During Concept Drift](https://mlsanity.com/paper/2412.09035/) (2024)
- [Time Series Analysis in Compressor-Based Machines: A Survey](https://mlsanity.com/paper/2402.17802/) (2024)
- [Global Lightning-Ignited Wildfires Prediction and Climate Change Projections based on Explainable Machine Learning Models](https://mlsanity.com/paper/2409.10046/) (2024)
- [SymboLLM-FE: LLM-Accelerated Symbolic Regression for Automated Feature Engineering on Tabular Data](https://mlsanity.com/paper/2608.28408/) (2026)

## Abstract (arXiv)

In the realm of machine and deep learning regression tasks, the role of effective feature engineering (FE) is pivotal in enhancing model performance. Traditional approaches of FE often rely on domain expertise to manually design features for machine learning models. In the context of deep learning models, the FE is embedded in the neural network's architecture, making it hard for interpretation. In this study, we propose to integrate symbolic regression (SR) as an FE process before a machine learning model to improve its performance. We show, through extensive experimentation on synthetic and real-world physics-related datasets, that the incorporation of SR-derived features significantly enhances the predictive capabilities of both machine and deep learning regression models with 34-86% root mean square error (RMSE) improvement in synthetic datasets and 4-11.5% improvement in real-world datasets. In addition, as a realistic use-case, we show the proposed method improves the machine learning performance in predicting superconducting critical temperatures based on Eliashberg theory by more than 20% in terms of RMSE. These results outline the potential of SR as an FE component in data-driven models.