mlsanityData to 7 Oct 2026

Sparse Weight Activation Training

arXiv 2001.01969 ↗First posted 7 Jan 2020cs.LG stat.MLcited by 45

Not on the map. Why?

Abstract (arXiv)

Neural network training is computationally and memory intensive. Sparse training can reduce the burden on emerging hardware platforms designed to accelerate sparse computations, but it can affect network convergence. In this work, we propose a novel CNN training algorithm Sparse Weight Activation Training (SWAT). SWAT is more computation and memory-efficient than conventional training. SWAT modifies back-propagation based on the empirical insight that convergence during training tends to be robust to the elimination of (i) small magnitude weights during the forward pass and (ii) both small magnitude weights and activations during the backward pass. We evaluate SWAT on recent CNN architectures such as ResNet, VGG, DenseNet and WideResNet using CIFAR-10, CIFAR-100 and ImageNet datasets. For ResNet-50 on ImageNet SWAT reduces total floating-point operations (FLOPS) during training by 80% resulting in a 3.3× training speedup when run on a simulated sparse learning accelerator representative of emerging platforms while incurring only 1.63% reduction in validation accuracy. Moreover, SWAT reduces memory footprint during the backward pass by 23% to 50% for activations and 50% to 90% for weights.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Jan 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 11; 2023: 9; 2024: 9; 2025: 3; 2026: 2.

45 papers.

Sort:

By area:

  1. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
  2. 2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 107
  3. 2021Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Modelscited by 75
  4. 2023SparseViT: Revisiting Activation Sparsity for Efficient High-Resolution Vision Transformercited by 60
  5. 2024Training-Free Activation Sparsity in Large Language Modelscited by 51
  6. 2021Dynamic Sparse Training for Deep Reinforcement Learningcited by 49
  7. 2022ZeroFL: Efficient On-Device Training for Federated Learning with Local Sparsitycited by 45
  8. 2021Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsitycited by 44
  9. 2022Sparsity Winning Twice: Better Robust Generalization from More Efficient Trainingcited by 41
  10. 2022Fine-tuning Language Models over Slow Networks using Activation Compression with Guaranteescited by 29
  11. 2021Selfish Sparse RNN Trainingcited by 26
  12. 2024Federated LoRA with Sparse Communicationcited by 26
33 more
  1. 2023Rethinking Graph Lottery Tickets: Graph Sparsity Matterscited by 23
  2. 2024A Survey of Lottery Ticket Hypothesiscited by 23
  3. 2021L2ight: Enabling On-Chip Learning for Optical Neural Networks via Efficient in-situ Subspace Optimizationcited by 21
  4. 2021Truly Sparse Neural Networks at Scalecited by 15
  5. 2022Where to Pay Attention in Sparse Training for Feature Selection?cited by 15
  6. 2024LSK3DNet: Towards Effective and Efficient 3D Perception with Large Sparse Kernelscited by 15
  7. 2022Lottery Aware Sparsity Hunting: Enabling Federated Learning on Resource-Limited Edgecited by 12
  8. 2023SparseProp: Efficient Sparse Backpropagation for Faster Training of Neural Networkscited by 12
  9. 2022Minimum Variance Unbiased N:M Sparsity for the Neural Gradientscited by 9
  10. 2024FedBAT: Communication-Efficient Federated Learning via Learnable Binarizationcited by 9
  11. 2025TeleSparse: Practical Privacy-Preserving Verification of Deep Neural Networkscited by 9
  12. 2022p-Meta: Towards On-device Deep Model Adaptationcited by 8
  13. 2022Safety and Performance, Why not Both? Bi-Objective Optimized Model Compression toward AI Software Deploymentcited by 8
  14. 2021Avoiding Forgetting and Allowing Forward Transfer in Continual Learning via Sparse Networkscited by 7
  15. 2023Sparse-IFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiencycited by 7
  16. 2023Training Large Language Models Efficiently with Sparsity and Dataflowcited by 7
  17. 2023Memory-adaptive Depth-wise Heterogeneous Federated Learningcited by 5
  18. 2022Diving into Unified Data-Model Sparsity for Class-Imbalanced Graph Representation Learningcited by 4
  19. 2024Safety and Performance, Why Not Both? Bi-Objective Optimized Model Compression against Heterogeneous Attacks Toward AI Software Deploymentcited by 4
  20. 2025FedRTS: Federated Robust Pruning via Combinatorial Thompson Samplingcited by 4
  21. 2021Search Spaces for Neural Model Trainingcited by 3
  22. 2023Compressing the Backward Pass of Large-Scale Neural Architectures by Structured Activation Pruningcited by 3
  23. 2024Efficient Backpropagation with Variance-Controlled Adaptive Samplingcited by 3
  24. 2022Exploiting the Partly Scratch-off Lottery Ticket for Quantization-Aware Trainingcited by 2
  25. 2023AUTOSPARSE: Towards Automated Sparse Training of Deep Neural Networkscited by 2
  26. 2023Federated learning compression designed for lightweight communicationscited by 2
  27. 2021SMOF: Squeezing More Out of Filters Yields Hardware-Friendly CNN Pruningcited by 1
  28. 2024Less Memory Means smaller GPUs: Backpropagation with Compressed Activationscited by 1
  29. 2022VeriCompress: A Tool to Streamline the Synthesis of Verified Robust Compressed Neural Networks from Scratchcited by 0
  30. 2024Dual sparse training framework: inducing activation map sparsity via Transformed ℓ1 regularizationcited by 0
  31. 2025Leveraging Label Potential for Enhanced Multimodal Emotion Recognitioncited by 0
  32. 2026HyperOffload: Graph-Driven Hierarchical Memory Management for Large Language Models on SuperNode Architecturescited by 0
  33. 2026Prox: Training-Free FFN Activation Sparsity via Approximate Intermediate-Channel Salience in LLMscited by 0