mlsanityData to 7 Oct 2026

Stochastic gradient descent with gradient estimator for categorical features

arXiv 2209.03771 ↗First posted 8 Sep 2022cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Categorical data are present in key areas such as health or supply chain, and this data require specific treatment. In order to apply recent machine learning models on such data, encoding is needed. In order to build interpretable models, one-hot encoding is still a very good solution, but such encoding creates sparse data. Gradient estimators are not suited for sparse data: the gradient is mainly considered as zero while it simply does not always exists, thus a novel gradient estimator is introduced. We show what this estimator minimizes in theory and show its efficiency on different datasets with multiple model architectures. This new estimator performs better than common estimators under similar settings. A real world retail dataset is also released after anonymization. Overall, the aim of this paper is to thoroughly consider categorical data and adapt models and optimizers to these key features.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 20 of its 45 references that are in the map. First posted on arXiv 8 Sep 2022.

Builds on

20 of 45 references.

References · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  5. 2021Revisiting Deep Learning Models for Tabular Datacited by 725
  6. 2019TabNet: Attentive Interpretable Tabular Learningcited by 640
  7. 2021Tabular Data: Deep Learning is Not All You Needcited by 530
  8. 2021Deep Neural Networks and Tabular Data: A Surveycited by 489
  9. 2018AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networkscited by 319
  10. 2017Distilling a Neural Network Into a Soft Decision Treecited by 230
  11. 2019Neural Oblivious Decision Ensembles for Deep Learning on Tabular Datacited by 200
  12. 2021Well-tuned Simple Nets Excel on Tabular Datasetscited by 151
8 more
  1. 2021Step-unrolled Denoising Autoencoders for Text Generationcited by 102
  2. 2017REBAR: Low-variance, unbiased gradient estimates for discrete latent variable modelscited by 78
  3. 2020Compression of Deep Learning Models for Text: A Surveycited by 64
  4. 2020Randomized Automatic Differentiationcited by 23
  5. 2021Unbiased Gradient Estimation with Balanced Assignments for Mixtures of Expertscited by 10
  6. 2016Stabilized Sparse Online Learning for Sparse Datacited by 2
  7. 2013Fast Learning from Sparse Datacited by 1
  8. 2016Linear Learning with Sparse Datacited by 1

Built on by

Not cited yet.