# GCWSNet: Generalized Consistent Weighted Sampling for Scalable and Accurate Training of Neural Networks

arXiv: https://arxiv.org/abs/2201.02283 · HTML: https://mlsanity.com/paper/2201.02283/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 63 references that are in the map.
- First posted on arXiv 7 Jan 2022.

## Builds on (most cited first, 7 of 7)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systems](https://mlsanity.com/paper/2003.05622/) (2020)
- [Fast Memory-efficient Anomaly Detection in Streaming Heterogeneous Graphs](https://mlsanity.com/paper/1602.04844/) (2016)
- [Preserving Semantic Neighborhoods for Robust Cross-modal Retrieval](https://mlsanity.com/paper/2007.08617/) (2020)
- [Locality-Sensitive Hashing Scheme based on Longest Circular Co-Substring](https://mlsanity.com/paper/2004.05345/) (2020)
- [Theory of the GMM Kernel](https://mlsanity.com/paper/1608.00550/) (2016)
- [Tunable GMM Kernels](https://mlsanity.com/paper/1701.02046/) (2017)

## Built on by (most cited first, 6 of 6)

- [OPORP: One Permutation + One Random Projection](https://mlsanity.com/paper/2302.03505/) (2023)
- [Differential Privacy with Random Projections and Sign Random Projections](https://mlsanity.com/paper/2306.01751/) (2023)
- [Package for Fast ABC-Boost](https://mlsanity.com/paper/2207.08770/) (2022)
- [CoopHash: Cooperative Learning of Multipurpose Descriptor and Contrastive Pair Generator via Variational MCMC Teaching for Supervised Image Hashing](https://mlsanity.com/paper/2210.04288/) (2022)
- [Noisy ℓ0-Sparse Subspace Clustering on Dimensionality Reduced Data](https://mlsanity.com/paper/2206.11079/) (2022)
- [pGMM Kernel Regression and Comparisons with Boosted Trees](https://mlsanity.com/paper/2207.08667/) (2022)

## Abstract (arXiv)

We develop the "generalized consistent weighted sampling" (GCWS) for hashing the "powered-GMM" (pGMM) kernel (with a tuning parameter p). It turns out that GCWS provides a numerically stable scheme for applying power transformation on the original data, regardless of the magnitude of p and the data. The power transformation is often effective for boosting the performance, in many cases considerably so. We feed the hashed data to neural networks on a variety of public classification datasets and name our method ``GCWSNet''. Our extensive experiments show that GCWSNet often improves the classification accuracy. Furthermore, it is evident from the experiments that GCWSNet converges substantially faster. In fact, GCWS often reaches a reasonable accuracy with merely (less than) one epoch of the training process. This property is much desired because many applications, such as advertisement click-through rate (CTR) prediction models, or data streams (i.e., data seen only once), often train just one epoch. Another beneficial side effect is that the computations of the first layer of the neural networks become additions instead of multiplications because the input data become binary (and highly sparse). Empirical comparisons with (normalized) random Fourier features (NRFF) are provided. We also propose to reduce the model size of GCWSNet by count-sketch and develop the theory for analyzing the impact of using count-sketch on the accuracy of GCWS. Our analysis shows that an ``8-bit'' strategy should work well in that we can always apply an 8-bit count-sketch hashing on the output of GCWS hashing without hurting the accuracy much. There are many other ways to take advantage of GCWS when training deep neural networks. For example, one can apply GCWS on the outputs of the last layer to boost the accuracy of trained deep neural networks.