# Regularized target encoding outperforms traditional methods in supervised machine learning with high cardinality features

arXiv: https://arxiv.org/abs/2104.00629 · HTML: https://mlsanity.com/paper/2104.00629/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 54 references that are in the map.
- First posted on arXiv 1 Apr 2021.

## Builds on (most cited first, 9 of 9)

- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [ranger: A Fast Implementation of Random Forests for High Dimensional Data in C++ and R](https://mlsanity.com/paper/1508.04409/) (2015)
- [Entity Embeddings of Categorical Variables](https://mlsanity.com/paper/1604.06737/) (2016)
- [Hyperparameters and Tuning Strategies for Random Forest](https://mlsanity.com/paper/1804.03515/) (2018)
- [Beyond One-hot Encoding: lower dimensional target embedding](https://mlsanity.com/paper/1806.10805/) (2018)
- [Similarity encoding for learning with dirty categorical variables](https://mlsanity.com/paper/1806.00979/) (2018)
- [Encoding high-cardinality string categorical variables](https://mlsanity.com/paper/1907.01860/) (2019)
- [liquidSVM: A Fast and Versatile SVM package](https://mlsanity.com/paper/1702.06899/) (2017)
- [Automatic Gradient Boosting](https://mlsanity.com/paper/1807.03873/) (2018)

## Built on by (most cited first, 21 of 21)

- [Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges](https://mlsanity.com/paper/2107.05847/) (2021)
- [Simplifying Polylogarithms with Machine Learning](https://mlsanity.com/paper/2206.04115/) (2022)
- [Fairness Implications of Encoding Protected Categorical Attributes](https://mlsanity.com/paper/2201.11358/) (2022)
- [Machine Learning with High-Cardinality Categorical Features in Actuarial Applications](https://mlsanity.com/paper/2301.12710/) (2023)
- [A benchmark of categorical encoders for binary classification](https://mlsanity.com/paper/2307.09191/) (2023)
- [Lessons from the AdKDD'21 Privacy-Preserving ML Challenge](https://mlsanity.com/paper/2201.13123/) (2022)
- [Encoding categorical data: Is there yet anything 'hotter' than one-hot encoding?](https://mlsanity.com/paper/2312.16930/) (2023)
- [Enabling Mixed Effects Neural Networks for Diverse, Clustered Data Using Monte Carlo Methods](https://mlsanity.com/paper/2407.01115/) (2024)
- [Interpretable vs Learned Encoders for High-Cardinality Fraud Detection](https://mlsanity.com/paper/2607.00477/) (2026)
- [Comparative Study on the Performance of Categorical Variable Encoders in Classification and Regression Tasks](https://mlsanity.com/paper/2401.09682/) (2024)
- [T-COL: Generating Counterfactual Explanations for General User Preferences on Variable Machine Learning Systems](https://mlsanity.com/paper/2309.16146/) (2023)
- [Predicting loss-of-function impact of genetic mutations: a machine learning approach](https://mlsanity.com/paper/2402.00054/) (2024)
- [Comparative Analysis of Data Preprocessing Methods, Feature Selection Techniques and Machine Learning Models for Improved Classification and Regression Performance on Imbalanced Genetic Data](https://mlsanity.com/paper/2402.14980/) (2024)
- [Methodology for an Analysis of Influencing Factors on 3D Object Detection Performance](https://mlsanity.com/paper/2411.08482/) (2024)
- [WildWood: a new Random Forest algorithm](https://mlsanity.com/paper/2109.08010/) (2021)
- [Introducing explainable supervised machine learning into interactive feedback loops for statistical production system](https://mlsanity.com/paper/2202.03212/) (2022)
- [Using Machine Learning Methods for Automation of Size Grid Building and Management](https://mlsanity.com/paper/2306.09775/) (2023)
- [Scalable Krylov Subspace Methods for Generalized Mixed-Effects Models with Crossed Random Effects](https://mlsanity.com/paper/2505.09552/) (2025)
- [SG-XDEAT: Sparsity-Guided Cross-Dimensional and Cross-Encoding Attention with Target-Aware Conditioning in Tabular Learning](https://mlsanity.com/paper/2510.12659/) (2025)
- [Robustness of AutoML on Dirty Categorical Data](https://mlsanity.com/paper/2602.00412/) (2026)
- [On the Encoding of Calendar Features in Covariate-Informed Time Series Foundation Models](https://mlsanity.com/paper/2610.13676/) (2026)

## Abstract (arXiv)

Since most machine learning (ML) algorithms are designed for numerical inputs, efficiently encoding categorical variables is a crucial aspect in data analysis. A common problem are high cardinality features, i.e. unordered categorical predictor variables with a high number of levels. We study techniques that yield numeric representations of categorical variables which can then be used in subsequent ML applications. We focus on the impact of these techniques on a subsequent algorithm's predictive performance, and -- if possible -- derive best practices on when to use which technique. We conducted a large-scale benchmark experiment, where we compared different encoding strategies together with five ML algorithms (lasso, random forest, gradient boosting, k-nearest neighbors, support vector machine) using datasets from regression, binary- and multiclass- classification settings. In our study, regularized versions of target encoding (i.e. using target predictions based on the feature levels in the training set as a new numerical feature) consistently provided the best results. Traditionally widely used encodings that make unreasonable assumptions to map levels to integers (e.g. integer encoding) or to reduce the number of levels (possibly based on target information, e.g. leaf encoding) before creating binary indicator variables (one-hot or dummy encoding) were not as effective in comparison.