# A Comparison of Machine Learning Methods for Data with High-Cardinality Categorical Variables

arXiv: https://arxiv.org/abs/2307.02071 · HTML: https://mlsanity.com/paper/2307.02071/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 21 references that are in the map.
- First posted on arXiv 5 Jul 2023.

## Builds on (most cited first, 7 of 7)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [Entity Embeddings of Categorical Variables](https://mlsanity.com/paper/1604.06737/) (2016)
- [Gaussian Process Boosting](https://mlsanity.com/paper/2004.02653/) (2020)
- [Integrating Random Effects in Deep Neural Networks](https://mlsanity.com/paper/2206.03314/) (2022)
- [Latent Gaussian Model Boosting](https://mlsanity.com/paper/2105.08966/) (2021)
- [Machine Learning with High-Cardinality Categorical Features in Actuarial Applications](https://mlsanity.com/paper/2301.12710/) (2023)

## Built on by (most cited first, 2 of 2)

- [Enabling Mixed Effects Neural Networks for Diverse, Clustered Data Using Monte Carlo Methods](https://mlsanity.com/paper/2407.01115/) (2024)
- [Scalable Krylov Subspace Methods for Generalized Mixed-Effects Models with Crossed Random Effects](https://mlsanity.com/paper/2505.09552/) (2025)

## Abstract (arXiv)

High-cardinality categorical variables are variables for which the number of different levels is large relative to the sample size of a data set, or in other words, there are few data points per level. Machine learning methods can have difficulties with high-cardinality variables. In this article, we empirically compare several versions of two of the most successful machine learning methods, tree-boosting and deep neural networks, and linear mixed effects models using multiple tabular data sets with high-cardinality categorical variables. We find that, first, machine learning models with random effects have higher prediction accuracy than their classical counterparts without random effects, and, second, tree-boosting with random effects outperforms deep neural networks with random effects.