# Similarity encoding for learning with dirty categorical variables

arXiv: https://arxiv.org/abs/1806.00979 · HTML: https://mlsanity.com/paper/1806.00979/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jun 2018.

## Built on by (most cited first, 28 of 28)

- [Regularized target encoding outperforms traditional methods in supervised machine learning with high cardinality features](https://mlsanity.com/paper/2104.00629/) (2021)
- [Fairness Implications of Encoding Protected Categorical Attributes](https://mlsanity.com/paper/2201.11358/) (2022)
- [Machine Learning with High-Cardinality Categorical Features in Actuarial Applications](https://mlsanity.com/paper/2301.12710/) (2023)
- [A benchmark of categorical encoders for binary classification](https://mlsanity.com/paper/2307.09191/) (2023)
- [Hybrid Parameter Search and Dynamic Model Selection for Mixed-Variable Bayesian Optimization](https://mlsanity.com/paper/2206.01409/) (2022)
- [Interpretable vs Learned Encoders for High-Cardinality Fraud Detection](https://mlsanity.com/paper/2607.00477/) (2026)
- [FedECA: Federated External Control Arms for Causal Inference with Time-To-Event Data in Distributed Settings](https://mlsanity.com/paper/2311.16984/) (2023)
- [Comparative Study on the Performance of Categorical Variable Encoders in Classification and Regression Tasks](https://mlsanity.com/paper/2401.09682/) (2024)
- [On Clustering Categories of Categorical Predictors in Generalized Linear Models](https://mlsanity.com/paper/2110.10059/) (2021)
- [ptype-cat: Inferring the Type and Values of Categorical Variables](https://mlsanity.com/paper/2111.11956/) (2021)
- [Identifying Substitute and Complementary Products for Assortment Optimization with Cleora Embeddings](https://mlsanity.com/paper/2208.06262/) (2022)
- [Predicting Treatment Adherence of Tuberculosis Patients at Scale](https://mlsanity.com/paper/2211.02943/) (2022)
- [Tabular Learning: Encoding for Entity and Context Embeddings](https://mlsanity.com/paper/2403.19405/) (2024)
- [YZS-model: A Predictive Model for Organic Drug Solubility Based on Graph Convolutional Networks and Transformer-Attention](https://mlsanity.com/paper/2406.19136/) (2024)
- [Using Text-Based Life Trajectories from Swedish Register Data to Predict Residential Mobility with Pretrained Transformers](https://mlsanity.com/paper/2512.07865/) (2025)
- [C2TC: A Training-Free Framework for Efficient Tabular Data Condensation](https://mlsanity.com/paper/2602.21717/) (2026)
- [Holmes: An Efficient and Lightweight Semantic Based Anomalous Email Detector](https://mlsanity.com/paper/2104.08044/) (2021)
- [From Strings to Data Science: a Practical Framework for Automated String Handling](https://mlsanity.com/paper/2111.01868/) (2021)
- [Performance Comparison of Simple Transformer and Res-CNN-BiLSTM for Cyberbullying Classification](https://mlsanity.com/paper/2206.02206/) (2022)
- [Progressive Feature Upgrade in Semi-supervised Learning on Tabular Domain](https://mlsanity.com/paper/2212.00892/) (2022)
- [From Data to Decisions: The Transformational Power of Machine Learning in Business Recommendations](https://mlsanity.com/paper/2402.08109/) (2024)
- [CLAMS: A System for Zero-Shot Model Selection for Clustering](https://mlsanity.com/paper/2407.11286/) (2024)
- [Data-driven Trust Bootstrapping for Mobile Edge Computing-based Industrial IoT Services](https://mlsanity.com/paper/2508.12560/) (2025)
- [Robustness of AutoML on Dirty Categorical Data](https://mlsanity.com/paper/2602.00412/) (2026)
- [Non-linear PCA via Evolution Strategies: a Novel Objective Function](https://mlsanity.com/paper/2602.03967/) (2026)
- [FlexTab: A Flexible Encoder-Decoder Architecture for In-Context Learning Across Diverse Tabular Tasks](https://mlsanity.com/paper/2606.30336/) (2026)
- [Enhancing the KidSat Model: Integrating Geographical Encoding and Data Quality Assessment for Childhood Poverty Prediction](https://mlsanity.com/paper/2607.08281/) (2026)
- [An Embarrassingly Simple Rule-based Visiting Circulation Approach to Trip Destination Prediction](https://mlsanity.com/paper/2607.25751/) (2026)

## Abstract (arXiv)

For statistical learning, categorical variables in a table are usually considered as discrete entities and encoded separately to feature vectors, e.g., with one-hot encoding. "Dirty" non-curated data gives rise to categorical variables with a very high cardinality but redundancy: several categories reflect the same entity. In databases, this issue is typically solved with a deduplication step. We show that a simple approach that exposes the redundancy to the learning algorithm brings significant gains. We study a generalization of one-hot encoding, similarity encoding, that builds feature vectors from similarities across categories. We perform a thorough empirical validation on non-curated tables, a problem seldom studied in machine learning. Results on seven real-world datasets show that similarity encoding brings significant gains in prediction in comparison with known encoding methods for categories or strings, notably one-hot encoding and bag of character n-grams. We draw practical recommendations for encoding dirty categories: 3-gram similarity appears to be a good choice to capture morphological resemblance. For very high-cardinality, dimensionality reduction significantly reduces the computational cost with little loss in performance: random projections or choosing a subset of prototype categories still outperforms classic encoding approaches.