mlsanityData to 7 Oct 2026

ResBit: Residual Bit Vector for Categorical Values

arXiv 2309.17196 ↗First posted 29 Sep 2023cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

One-hot vectors, a common method for representing discrete/categorical data, in machine learning are widely used because of their simplicity and intuitiveness. However, one-hot vectors suffer from a linear increase in dimensionality, posing computational and memory challenges, especially when dealing with datasets containing numerous categories. In this paper, we focus on tabular data generation, and reveal the multinomial diffusion faces the mode collapse phenomenon when the cardinality is high. Moreover, due to the limitations of one-hot vectors, the training phase takes time longer in such a situation. To address these issues, we propose Residual Bit Vectors (ResBit), a technique for densely representing categorical data. ResBit is an extension of analog bits and overcomes limitations of analog bits when applied to tabular data generation. Our experiments demonstrate that ResBit not only accelerates training but also maintains performance when compared with the situations before applying ResBit. Furthermore, our results indicate that many existing methods struggle with high-cardinality data, underscoring the need for lower-dimensional representations, such as ResBit and latent vectors.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 47 of its 68 references that are in the map. First posted on arXiv 29 Sep 2023.

Builds on

47 of 68 references.

References · 47

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2020Scaling Laws for Neural Language Modelscited by 6,411
  12. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
35 more
  1. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  2. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  3. 2016Improved Techniques for Training GANscited by 2,938
  4. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  5. 2017Improved Training of Wasserstein GANscited by 2,223
  6. 2014Conditional Generative Adversarial Netscited by 2,152
  7. 2019Searching for MobileNetV3cited by 1,984
  8. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  9. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  10. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  11. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  12. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  13. 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 858
  14. 2019Modeling Tabular data using Conditional GANcited by 695
  15. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  16. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 566
  17. 2021Tackling the Generative Learning Trilemma with Denoising Diffusion GANscited by 515
  18. 2022DiGress: Discrete Denoising diffusion for graph generationcited by 503
  19. 2016SeqGAN: Sequence Generative Adversarial Nets with Policy Gradientcited by 375
  20. 2022TabDDPM: Modelling Tabular Data with Diffusion Modelscited by 373
  21. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioningcited by 362
  22. 2017Real-valued (Medical) Time Series Generation with Recurrent Conditional GANscited by 329
  23. 2021How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Modelscited by 193
  24. 2021CTAB-GAN: Effective Table Data Synthesizingcited by 156
  25. 2023RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Pathscited by 141
  26. 2019GANSynth: Adversarial Neural Audio Synthesiscited by 114
  27. 2017Long Text Generation via Adversarial Training with Leaked Informationcited by 100
  28. 2018Synthesizing Tabular Data using Generative Adversarial Networkscited by 92
  29. 2023Generative Time Series Forecasting with Diffusion, Denoise, and Disentanglementcited by 89
  30. 2022STaSy: Score-based Tabular data Synthesiscited by 85
  31. 2023CoDi: Co-evolving Contrastive Diffusion Models for Mixed-type Tabular Synthesiscited by 78
  32. 2022HouseDiffusion: Vector Floorplan Generation via a Diffusion Model with Discrete and Continuous Denoisingcited by 72
  33. 2021PSA-GAN: Progressive Self Attention GANs for Synthetic Time Seriescited by 58
  34. 2021Towards Generating Real-World Time Series Datacited by 26
  35. 2022Collapse by Conditioning: Training Class-conditional GANs with Limited Datacited by 24

Built on by

Not cited yet.