# Scalable and Sustainable Deep Learning via Randomized Hashing

arXiv: https://arxiv.org/abs/1602.08194 · HTML: https://mlsanity.com/paper/1602.08194/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Feb 2016.

## Built on by (most cited first, 17 of 17)

- [Multiplying Matrices Without Multiplying](https://mlsanity.com/paper/2106.10860/) (2021)
- [Accelerating SLIDE Deep Learning on Modern CPUs: Vectorization, Quantizations, Memory Optimizations, and More](https://mlsanity.com/paper/2103.10891/) (2021)
- [Sketch-GNN: Scalable Graph Neural Networks with Sublinear Training Complexity](https://mlsanity.com/paper/2406.15575/) (2024)
- [NoMAD-Attention: Efficient LLM Inference on CPUs Through Multiply-add-free Attention](https://mlsanity.com/paper/2403.01273/) (2024)
- [Linear Bandit Algorithms with Sublinear Time Complexity](https://mlsanity.com/paper/2103.02729/) (2021)
- [In defense of parameter sharing for model-compression](https://mlsanity.com/paper/2310.11611/) (2023)
- [Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions](https://mlsanity.com/paper/2108.13055/) (2021)
- [Distributed SLIDE: Enabling Training Large Neural Networks on Low Bandwidth and Simple CPU-Clusters via Model Parallelism and Sparsity](https://mlsanity.com/paper/2201.12667/) (2022)
- [BOLT: An Automated Deep Learning Framework for Training and Deploying Large-Scale Search and Recommendation Models on Commodity CPU Hardware](https://mlsanity.com/paper/2303.17727/) (2023)
- [Large-Scale Distributed Learning via Private On-Device Locality-Sensitive Hashing](https://mlsanity.com/paper/2306.02563/) (2023)
- [Stella Nera: A Differentiable Maddness-Based Hardware Accelerator for Efficient Approximate Matrix Multiplication](https://mlsanity.com/paper/2311.10207/) (2023)
- [Dynamic Network Adaptation at Inference](https://mlsanity.com/paper/2204.08400/) (2022)
- [[Experiments & Analysis] Evaluating the Feasibility of Sampling-Based Techniques for Training Multilayer Perceptrons](https://mlsanity.com/paper/2306.09293/) (2023)
- [Distance Adaptive Beam Search for Provably Accurate Graph-Based Nearest Neighbor Search](https://mlsanity.com/paper/2505.15636/) (2025)
- [OneShot: Index-in-Ranking with Neural Scoring for Large-Scale Retrieval](https://mlsanity.com/paper/2607.27475/) (2026)
- [LeanStream: A Speculate-and-Refine Streaming Framework for Efficient on-Device LLM Inference](https://mlsanity.com/paper/2609.03079/) (2026)
- [Trustworthy Data- and ML-Ops for Intelligent Transportation Systems and Logistics](https://mlsanity.com/paper/2610.01282/) (2026)

## Abstract (arXiv)

Current deep learning architectures are growing larger in order to learn from complex datasets. These architectures require giant matrix multiplication operations to train millions of parameters. Conversely, there is another growing trend to bring deep learning to low-power, embedded devices. The matrix operations, associated with both training and testing of deep networks, are very expensive from a computational and energy standpoint. We present a novel hashing based technique to drastically reduce the amount of computation needed to train and test deep networks. Our approach combines recent ideas from adaptive dropouts and randomized hashing for maximum inner product search to select the nodes with the highest activation efficiently. Our new algorithm for deep learning reduces the overall computational cost of forward and back-propagation by operating on significantly fewer (sparse) nodes. As a consequence, our algorithm uses only 5% of the total multiplications, while keeping on average within 1% of the accuracy of the original model. A unique property of the proposed hashing based back-propagation is that the updates are always sparse. Due to the sparse gradient updates, our algorithm is ideally suited for asynchronous and parallel training leading to near linear speedup with increasing number of cores. We demonstrate the scalability and sustainability (energy efficiency) of our proposed algorithm via rigorous experimental evaluations on several real datasets.