# Procrustes: a Dataflow and Accelerator for Sparse Deep Neural Network Training

arXiv: https://arxiv.org/abs/2009.10976 · HTML: https://mlsanity.com/paper/2009.10976/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Sep 2020.

## Built on by (most cited first, 18 of 18)

- [Sparse Training via Boosting Pruning Plasticity with Neuroregeneration](https://mlsanity.com/paper/2106.10404/) (2021)
- [Sparsity Winning Twice: Better Robust Generalization from More Efficient Training](https://mlsanity.com/paper/2202.09844/) (2022)
- [FAST: DNN Training Under Variable Precision Block Floating Point with Stochastic Rounding](https://mlsanity.com/paper/2110.15456/) (2021)
- [Sparseloop: An Analytical Approach To Sparse Tensor Accelerator Modeling](https://mlsanity.com/paper/2205.05826/) (2022)
- [SparseProp: Efficient Sparse Backpropagation for Faster Training of Neural Networks](https://mlsanity.com/paper/2302.04852/) (2023)
- [SAFT: Towards Out-of-Distribution Generalization in Fine-Tuning](https://mlsanity.com/paper/2407.03036/) (2024)
- [FlexBlock: A Flexible DNN Training Accelerator with Multi-Mode Block Floating Point Support](https://mlsanity.com/paper/2203.06673/) (2022)
- [Accelerating DNN Training with Structured Data Gradient Pruning](https://mlsanity.com/paper/2202.00774/) (2022)
- [MCBP: A Memory-Compute Efficient LLM Inference Accelerator Leveraging Bit-Slice-enabled Sparsity and Repetitiveness](https://mlsanity.com/paper/2509.10372/) (2025)
- [Shift-BNN: Highly-Efficient Probabilistic Bayesian Neural Network Training via Memory-Friendly Pattern Retrieving](https://mlsanity.com/paper/2110.03553/) (2021)
- [MERCURY: Accelerating DNN Training By Exploiting Input Similarity](https://mlsanity.com/paper/2110.14904/) (2021)
- [CAMEL: Co-Designing AI Models and Embedded DRAMs for Efficient On-Device Learning](https://mlsanity.com/paper/2305.03148/) (2023)
- [LearningGroup: A Real-Time Sparse Training on FPGA via Learnable Weight Grouping for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2210.16624/) (2022)
- [YFlows: Systematic Dataflow Exploration and Code Generation for Efficient Neural Network Inference using SIMD Architectures on CPUs](https://mlsanity.com/paper/2310.00574/) (2023)
- [BIM: Block-Wise Self-Supervised Learning with Masked Image Modeling](https://mlsanity.com/paper/2311.17218/) (2023)
- [Entropy Meets Importance: A Unified Head Importance-Entropy Score for Stable and Efficient Transformer Pruning](https://mlsanity.com/paper/2510.13832/) (2025)
- [Dynamic Neural Network Architectural and Topological Adaptation and Related Methods -- A Survey](https://mlsanity.com/paper/2108.10066/) (2021)
- [TinyCL: An Efficient Hardware Architecture for Continual Learning on Autonomous Systems](https://mlsanity.com/paper/2402.09780/) (2024)

## Abstract (arXiv)

The success of DNN pruning has led to the development of energy-efficient inference accelerators that support pruned models with sparse weight and activation tensors. Because the memory layouts and dataflows in these architectures are optimized for the access patterns during mathitinference, however, they do not efficiently support the emerging sparse mathittraining techniques. In this paper, we demonstrate (a) that accelerating sparse training requires a co-design approach where algorithms are adapted to suit the constraints of hardware, and (b) that hardware for sparse DNN training must tackle constraints that do not arise in inference accelerators. As proof of concept, we adapt a sparse training algorithm to be amenable to hardware acceleration; we then develop dataflow, data layout, and load-balancing techniques to accelerate it. The resulting system is a sparse DNN training accelerator that produces pruned models with the same accuracy as dense models without first training, then pruning, and finally retraining, a dense model. Compared to training the equivalent unpruned models using a state-of-the-art DNN accelerator without sparse training support, Procrustes consumes up to 3.26× less energy and offers up to 4× speedup across a range of models, while pruning weights by an order of magnitude and maintaining unpruned accuracy.