# The Impact of Feature Scaling In Machine Learning: Effects on Regression and Classification Tasks

arXiv: https://arxiv.org/abs/2506.08274 · HTML: https://mlsanity.com/paper/2506.08274/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 133 references that are in the map.
- First posted on arXiv 9 Jun 2025.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [TabNet: Attentive Interpretable Tabular Learning](https://mlsanity.com/paper/1908.07442/) (2019)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation](https://mlsanity.com/paper/2010.16061/) (2020)
- [Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning](https://mlsanity.com/paper/1811.12808/) (2018)
- [Transfer Learning with Deep Tabular Models](https://mlsanity.com/paper/2206.15306/) (2022)
- [Machine Learning Advances for Time Series Forecasting](https://mlsanity.com/paper/2012.12802/) (2020)
- [A Closer Look at Deep Learning Methods on Tabular Datasets](https://mlsanity.com/paper/2407.00956/) (2024)
- [Reproducibility in Machine Learning-based Research: Overview, Barriers and Drivers](https://mlsanity.com/paper/2406.14325/) (2024)
- [Graph Machine Learning in the Era of Large Language Models (LLMs)](https://mlsanity.com/paper/2404.14928/) (2024)
- [Sources of Irreproducibility in Machine Learning: A Review](https://mlsanity.com/paper/2204.07610/) (2022)
- [CatBoost: unbiased boosting with categorical features](https://mlsanity.com/paper/1706.09516/) (2017)
- [Clustering Algorithms: A Comparative Approach](https://mlsanity.com/paper/1612.08388/) (2016)
- [Reproducibility in Machine Learning for Health](https://mlsanity.com/paper/1907.01463/) (2019)
- [The choice of scaling technique matters for classification performance](https://mlsanity.com/paper/2212.12343/) (2022)
- [Optimal Ratio for Data Splitting](https://mlsanity.com/paper/2202.03326/) (2022)
- [Reproducibility in Machine Learning-Driven Research](https://mlsanity.com/paper/2307.10320/) (2023)
- [Machine Learning in Wireless Sensor Networks: Algorithms, Strategies, and Applications](https://mlsanity.com/paper/1405.4463/) (2014)
- [Leaf Classification Using Shape, Color, and Texture Features](https://mlsanity.com/paper/1401.4447/) (2013)
- [PCA, SVD, and Centering of Data](https://mlsanity.com/paper/2307.15213/) (2023)
- [Robust Statistical Scaling of Outlier Scores: Improving the Quality of Outlier Probabilities for Outliers (Extended Version)](https://mlsanity.com/paper/2408.15874/) (2024)

## Built on by (most cited first, 4 of 4)

- [Anomaly detection in network flows using unsupervised online machine learning](https://mlsanity.com/paper/2509.01375/) (2025)
- [A Compound Classification System Based on Fuzzy Relations Applied to the Noise-Tolerant Control of a Bionic Hand via EMG Signal Recognition](https://mlsanity.com/paper/2509.20523/) (2025)
- [Using machine learning methods to predict cognitive age from psychophysiological tests](https://mlsanity.com/paper/2511.00013/) (2025)
- [Breaking the Homogeneity Assumption: Specialized Multi-Generator Adversarial Learning for Rare Failure Detection in Predictive Maintenance](https://mlsanity.com/paper/2607.19153/) (2026)

## Abstract (arXiv)

This research addresses the critical lack of comprehensive studies on feature scaling by systematically evaluating 12 scaling techniques - including several less common transformations - across 14 different Machine Learning algorithms and 16 datasets for classification and regression tasks. We meticulously analyzed impacts on predictive performance (using metrics such as accuracy, MAE, MSE, and R2) and computational costs (training time, inference time, and memory usage). Key findings reveal that while ensemble methods (such as Random Forest and gradient boosting models like XGBoost, CatBoost and LightGBM) demonstrate robust performance largely independent of scaling, other widely used models such as Logistic Regression, SVMs, TabNet, and MLPs show significant performance variations highly dependent on the chosen scaler. This extensive empirical analysis, with all source code, experimental results, and model parameters made publicly available to ensure complete transparency and reproducibility, offers model-specific crucial guidance to practitioners on the need for an optimal selection of feature scaling techniques.