# Sparse evolutionary Deep Learning with over one million artificial neurons on commodity hardware

arXiv: https://arxiv.org/abs/1901.09181 · HTML: https://mlsanity.com/paper/1901.09181/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jan 2019.

## Built on by (most cited first, 30 of 48)

- [Chasing Sparsity in Vision Transformers: An End-to-End Exploration](https://mlsanity.com/paper/2106.04533/) (2021)
- [More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity](https://mlsanity.com/paper/2207.03620/) (2022)
- [Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training](https://mlsanity.com/paper/2102.02887/) (2021)
- [The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training](https://mlsanity.com/paper/2202.02643/) (2022)
- [Sparse Training via Boosting Pruning Plasticity with Neuroregeneration](https://mlsanity.com/paper/2106.10404/) (2021)
- [Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs](https://mlsanity.com/paper/2310.08915/) (2023)
- [Dynamic Sparse Training for Deep Reinforcement Learning](https://mlsanity.com/paper/2106.04217/) (2021)
- [Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity](https://mlsanity.com/paper/2106.14568/) (2021)
- [NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networks](https://mlsanity.com/paper/2206.09117/) (2022)
- [Achieving Personalized Federated Learning with Sparse Local Models](https://mlsanity.com/paper/2201.11380/) (2022)
- [Selfish Sparse RNN Training](https://mlsanity.com/paper/2101.09048/) (2021)
- [Ten Lessons We Have Learned in the New "Sparseland": A Short Handbook for Sparse Neural Network Researchers](https://mlsanity.com/paper/2302.02596/) (2023)
- [Dynamic Sparsity Is Channel-Level Sparsity Learner](https://mlsanity.com/paper/2305.19454/) (2023)
- [Fantastic Weights and How to Find Them: Where to Prune in Dynamic Sparse Training](https://mlsanity.com/paper/2306.12230/) (2023)
- [Truly Sparse Neural Networks at Scale](https://mlsanity.com/paper/2102.01732/) (2021)
- [E2ENet: Dynamic Sparse Feature Fusion for Accurate and Efficient 3D Medical Image Segmentation](https://mlsanity.com/paper/2312.04727/) (2023)
- [A Bregman Learning Framework for Sparse Neural Networks](https://mlsanity.com/paper/2105.04319/) (2021)
- [Where to Pay Attention in Sparse Training for Feature Selection?](https://mlsanity.com/paper/2211.14627/) (2022)
- [LSK3DNet: Towards Effective and Efficient 3D Perception with Large Sparse Kernels](https://mlsanity.com/paper/2403.15173/) (2024)
- [You Can Have Better Graph Neural Networks by Not Training Weights at All: Finding Untrained GNNs Tickets](https://mlsanity.com/paper/2211.15335/) (2022)
- [Sparse Training Theory for Scalable and Efficient Agents](https://mlsanity.com/paper/2103.01636/) (2021)
- [Automatic Noise Filtering with Dynamic Sparse Training in Deep Reinforcement Learning](https://mlsanity.com/paper/2302.06548/) (2023)
- [Towards Structured Dynamic Sparse Pre-Training of BERT](https://mlsanity.com/paper/2108.06277/) (2021)
- [Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Survey](https://mlsanity.com/paper/2205.08099/) (2022)
- [Lottery Aware Sparsity Hunting: Enabling Federated Learning on Resource-Limited Edge](https://mlsanity.com/paper/2208.13092/) (2022)
- [Dynamic Sparse Training versus Dense Training: The Unexpected Winner in Image Corruption Robustness](https://mlsanity.com/paper/2410.03030/) (2024)
- [On the Neural Tangent Kernel Analysis of Randomly Pruned Neural Networks](https://mlsanity.com/paper/2203.14328/) (2022)
- [Safety and Performance, Why not Both? Bi-Objective Optimized Model Compression toward AI Software Deployment](https://mlsanity.com/paper/2208.05969/) (2022)
- [Don't Be So Dense: Sparse-to-Sparse GAN Training Without Sacrificing Performance](https://mlsanity.com/paper/2203.02770/) (2022)
- [Superposing Many Tickets into One: A Performance Booster for Sparse Neural Network Training](https://mlsanity.com/paper/2205.15322/) (2022)

## Abstract (arXiv)

Artificial Neural Networks (ANNs) have emerged as hot topics in the research community. Despite the success of ANNs, it is challenging to train and deploy modern ANNs on commodity hardware due to the ever-increasing model size and the unprecedented growth in the data volumes. Particularly for microarray data, the very-high dimensionality and the small number of samples make it difficult for machine learning techniques to handle. Furthermore, specialized hardware such as Graphics Processing Unit (GPU) is expensive. Sparse neural networks are the leading approaches to address these challenges. However, off-the-shelf sparsity inducing techniques either operate from a pre-trained model or enforce the sparse structure via binary masks. The training efficiency of sparse neural networks cannot be obtained practically. In this paper, we introduce a technique allowing us to train truly sparse neural networks with fixed parameter count throughout training. Our experimental results demonstrate that our method can be applied directly to handle high dimensional data, while achieving higher accuracy than the traditional two phases approaches. Moreover, we have been able to create truly sparse MultiLayer Perceptrons (MLPs) models with over one million neurons and to train them on a typical laptop without GPU, this being way beyond what is possible with any state-of-the-art techniques.