mlsanityData to 7 Oct 2026

PruneTrain: Fast Neural Network Training by Dynamic Sparse Model Reconfiguration

arXiv 1901.09290 ↗First posted 26 Jan 2019cs.LG stat.MLcited by 41

Not on the map. Why?

Abstract (arXiv)

State-of-the-art convolutional neural networks (CNNs) used in vision applications have large models with numerous weights. Training these models is very compute- and memory-resource intensive. Much research has been done on pruning or compressing these models to reduce the cost of inference, but little work has addressed the costs of training. We focus precisely on accelerating training. We propose PruneTrain, a cost-efficient mechanism that gradually reduces the training cost during training. PruneTrain uses a structured group-lasso regularization approach that drives the training optimization toward both high accuracy and small weight values. Small weights can then be periodically removed by reconfiguring the network model to a smaller one. By using a structured-pruning approach and additional reconfiguration techniques we introduce, the pruned model can still be efficiently processed on a GPU accelerator. Overall, PruneTrain achieves a reduction of 39% in the end-to-end training time of ResNet50 for ImageNet by reducing computation cost by 40% in FLOPs, memory accesses by 37% for memory bandwidth bound layers, and the inter-accelerator communication by 55%.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Jan 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 8; 2023: 7; 2024: 5; 2025: 6; 2026: 2.

41 papers.

Sort:

  1. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
  2. 2023A Comprehensive Survey of Dataset Distillationcited by 114
  3. 2021MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edgecited by 77
  4. 2021Only Train Once: A One-Shot Neural Network Training And Pruning Frameworkcited by 70
  5. 2022Deep Differentiable Logic Gate Networkscited by 67
  6. 2022Structural Pruning via Latency-Saliency Knapsackcited by 40
  7. 2021When to Prune? A Policy towards Early Structural Pruningcited by 38
  8. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  9. 2022CHEX: CHannel EXploration for CNN Model Compressioncited by 36
  10. 2023OTOV2: Automatic, Generic, User-Friendlycited by 32
  11. 2021Efficient Neural Network Training via Forward and Backward Propagation Sparsificationcited by 30
  12. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
29 more
  1. 2021Effective Model Sparsification by Scheduled Grow-and-Prune Methodscited by 25
  2. 2023Ten Lessons We Have Learned in the New "Sparseland": A Short Handbook for Sparse Neural Network Researcherscited by 25
  3. 2022Winning the Lottery Ahead of Time: Efficient Early Network Pruningcited by 21
  4. 2023ElasticTrainer: Speeding Up On-Device Training with Runtime Elastic Tensor Selectioncited by 12
  5. 2022Layer Freezing & Data Sieving: Missing Pieces of a Generic Framework for Sparse Trainingcited by 11
  6. 2024ELRT: Efficient Low-Rank Training for Compact Convolutional Neural Networkscited by 11
  7. 2021HALP: Hardware-Aware Latency Pruningcited by 8
  8. 2021GST: Group-Sparse Training for Accelerating Deep Reinforcement Learningcited by 7
  9. 2021AutoScore-Imbalance: An interpretable machine learning tool for development of clinical scores with rare events datacited by 6
  10. 2021Local Critic Training for Model-Parallel Learning of Deep Neural Networkscited by 5
  11. 2024Maxwell's Demon at Work: Efficient Pruning by Leveraging Saturation of Neuronscited by 5
  12. 2021AdaptCL: Efficient Collaborative Learning with Dynamic and Adaptive Pruningcited by 4
  13. 2022Scientometric Review of Artificial Intelligence for Operations & Maintenance of Wind Turbines: The Past, Present and Futurecited by 4
  14. 2022FedDCT: Federated Learning of Large Convolutional Neural Networks on Resource Constrained Devices using Divide and Collaborative Trainingcited by 3
  15. 2024DRIVE: Dual Gradient-Based Rapid Iterative Pruningcited by 3
  16. 2021An Oracle for Guiding Large-Scale Model/Hybrid Parallel Training of Convolutional Neural Networkscited by 2
  17. 2023FedDIP: Federated Learning with Extreme Dynamic Pruning and Incremental Regularizationcited by 2
  18. 2024SwitchLoRA: Switched Low-Rank Adaptation Can Learn Full-Rank Informationcited by 2
  19. 2023STen: Productive and Efficient Sparsity in PyTorchcited by 1
  20. 2024Decay Pruning Method: Smooth Pruning With a Self-Rectifying Procedurecited by 1
  21. 2025Advancing Weight and Channel Sparsification with Enhanced Saliencycited by 1
  22. 2025AdaBet: Gradient-free Layer Selection for Efficient Training of Deep Neural Networkscited by 1
  23. 2025Optimizing Storage Overhead of User Behavior Log for ML-embedded Mobile Appscited by 1
  24. 2025Theoretical Compression Bounds for Wide Multilayer Perceptronscited by 1
  25. 2021DANCE: DAta-Network Co-optimization for Efficient Segmentation Model Training and Inferencecited by 0
  26. 2025Loss-Aware Automatic Selection of Structured Pruning Criteria for Deep Neural Network Accelerationcited by 0
  27. 2025Neuro-inspired Ensemble-to-Ensemble Communication Primitives for Sparse and Efficient ANNscited by 0
  28. 2026Optimizing Feature Extraction for On-device Model Inference with User Behavior Sequencescited by 0
  29. 2026Squeeze-Release: Iterative Pruning with Exact Structural Minimizationcited by 0