mlsanityData to 7 Oct 2026

meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting

arXiv 1706.06197 ↗First posted 19 Jun 2017cs.LG cs.AI cs.CLcited by 34

Not on the map. Why?

Abstract (arXiv)

We propose a simple yet effective technique for neural network learning. The forward propagation is computed as usual. In back propagation, only a small subset of the full gradient is computed to update the model parameters. The gradient vectors are sparsified in such a way that only the top-k elements (in terms of magnitude) are kept. As a result, only k rows or columns (depending on the layout) of the weight matrix are modified, leading to a linear reduction (k divided by the vector dimension) in the computational cost. Surprisingly, experimental results demonstrate that we can update only 1-4% of the weights at each back propagation pass. This does not result in a larger number of training iterations. More interestingly, the accuracy of the resulting models is actually improved rather than degraded, and a detailed analysis is given. The code is available at https://github.com/lancopku/meProp

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Jun 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 7; 2023: 8; 2024: 8; 2025: 3; 2026: 1.

34 papers.

Sort:

By area:

  1. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  2. 2023A Comprehensive Survey of Dataset Distillationcited by 114
  3. 2023Towards Accurate Post-Training Quantization for Vision Transformercited by 70
  4. 2021M-FAC: Efficient Matrix-Free Approximations of Second-Order Informationcited by 54
  5. 2024Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Modelscited by 50
  6. 2022Beyond Explaining: Opportunities and Challenges of XAI-Based Model Improvementcited by 48
  7. 2022ZeroFL: Efficient On-Device Training for Federated Learning with Local Sparsitycited by 45
  8. 2024Gradient Routing: Masking Gradients to Localize Computation in Neural Networkscited by 28
  9. 2024Zeroth-Order Fine-Tuning of LLMs in Random Subspacescited by 23
  10. 2021L2ight: Enabling On-Chip Learning for Optical Neural Networks via Efficient in-situ Subspace Optimizationcited by 21
  11. 2021Masked Training of Neural Networks with Partial Gradientscited by 15
  12. 2022Optimization with Access to Auxiliary Informationcited by 12
22 more
  1. 2022Structured Pruning Adapterscited by 12
  2. 2023ElasticTrainer: Speeding Up On-Device Training with Runtime Elastic Tensor Selectioncited by 12
  3. 2022Minimum Variance Unbiased N:M Sparsity for the Neural Gradientscited by 9
  4. 2021Toward Compact Deep Neural Networks via Energy-Aware Pruningcited by 6
  5. 2021Where do Models go Wrong? Parameter-Space Saliency Maps for Explainabilitycited by 6
  6. 2024Data Reconstruction Attacks and Defenses: A Systematic Evaluationcited by 5
  7. 2024Diversity-aware Channel Pruning for StyleGAN Compressioncited by 5
  8. 2021Dynamic Collective Intelligence Learning: Finding Efficient Sparse Model via Refined Gradients for Pruned Weightscited by 4
  9. 2023TinyProp -- Adaptive Sparse Backpropagation for Efficient TinyML On-device Learningcited by 4
  10. 2024Advancing On-Device Neural Network Training with TinyPropv2: Dynamic, Sparse, and Efficient Backpropagationcited by 3
  11. 2025FedEL: Federated Elastic Learning for Heterogeneous Devicescited by 3
  12. 2021Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directionscited by 2
  13. 2022Exploiting the Partly Scratch-off Lottery Ticket for Quantization-Aware Trainingcited by 2
  14. 2023Straggler-resilient Federated Learning: Tackling Computation Heterogeneity with Layer-wise Partial Model Training in Mobile Edge Networkcited by 2
  15. 2025Training Large Neural Networks With Low-Dimensional Error Feedbackcited by 2
  16. 2022Aligned Weight Regularizers for Pruning Pretrained Neural Networkscited by 1
  17. 2024SparseGrad: A Selective Method for Efficient Fine-tuning of MLP Layerscited by 1
  18. 2023Efficient Online Processing with Deep Neural Networkscited by 0
  19. 2023Understanding Parameter Saliency via Extreme Value Theorycited by 0
  20. 2024ssProp: Energy-Efficient Training for Convolutional Neural Networks with Scheduled Sparse Back Propagationcited by 0
  21. 2025AMAQ: Adaptive Mixed-bit Activation Quantization for Collaborative Parameter Efficient Fine-tuningcited by 0
  22. 2026Unbiased Approximate Vector-Jacobian Products for Efficient Backpropagationcited by 0