mlsanityData to 14 Oct 2026

Masked Training of Neural Networks with Partial Gradients

arXiv 2106.08895 ↗First posted 16 Jun 2021cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

State-of-the-art training algorithms for deep learning models are based on stochastic gradient descent (SGD). Recently, many variations have been explored: perturbing parameters for better accuracy (such as in Extragradient), limiting SGD updates to a subset of parameters for increased efficiency (such as meProp) or a combination of both (such as Dropout). However, the convergence of these methods is often not studied in theory. We propose a unified theoretical framework to study such SGD variants -- encompassing the aforementioned algorithms and additionally a broad variety of methods used for communication efficient training or model compression. Our insights can be used as a guide to improve the efficiency of such methods and facilitate generalization to new applications. As an example, we tackle the task of jointly training networks, a version of which (limited to sub-networks) is used to create Slimmable Networks. By training a low-rank Transformer jointly with a standard one we obtain superior performance than when it is trained separately.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 35 of its 52 references that are in the map. First posted on arXiv 16 Jun 2021.

Builds on

35 of 52 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  4. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  5. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  6. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  7. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  8. 2018On First-Order Meta-Learning Algorithmscited by 1,018
  9. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 787
  10. 2016Wide & Deep Learning for Recommender Systemscited by 747
  11. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  12. 2017DeepFM: A Factorization-Machine based Neural Network for CTR Predictioncited by 676
23 more
  1. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 637
  2. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 603
  3. 2018Sparsified SGD with Memorycited by 376
  4. 2018Slimmable Neural Networkscited by 305
  5. 2017Sparse Communication for Distributed Gradient Descentcited by 249
  6. 2018The Convergence of Sparsified Gradient Methodscited by 229
  7. 2019Universally Slimmable Networks and Improved Training Techniquescited by 221
  8. 2020BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Modelscited by 188
  9. 2020Characterising Bias in Compressed Modelscited by 167
  10. 2020Dynamic Model Pruning with Feedbackcited by 125
  11. 2019Evaluating the Search Phase of Neural Architecture Searchcited by 120
  12. 2020On the Convergence of SGD with Biased Gradientscited by 83
  13. 2019Unified Optimal Analysis of the (Stochastic) Gradient Methodcited by 82
  14. 2021Pufferfish: Communication-efficient Models At No Extra Costcited by 61
  15. 2018A Tight Convergence Analysis for Stochastic Gradient Descent with Delayed Updatescited by 47
  16. 2017meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfittingcited by 34
  17. 2017Coordinating Filters for Faster Deep Neural Networkscited by 24
  18. 2020Dynamic Sparsity Neural Networks for Automatic Speech Recognitioncited by 23
  19. 2021Critical Parameters for Scalable Distributed Learning with Large Batches and Asynchronous Updatescited by 17
  20. 2016Training Skinny Deep Neural Networks with Iterative Hard Thresholding Methodscited by 16
  21. 2019On the Convergence of (Stochastic) Gradient Descent with Extrapolation for Non-Convex Optimizationcited by 6
  22. 2020Almost Sure Convergence of Dropout Algorithms for Neural Networkscited by 5
  23. 2020Asymptotic convergence rate of Dropout on shallow linear neural networkscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 6; 2024: 4; 2025: 3; 2026: 1.

16 papers.

Sort:

By area:

  1. 2024Gradient Routing: Masking Gradients to Localize Computation in Neural Networkscited by 34
  2. 2022Sparse Random Networks for Communication-Efficient Federated Learningcited by 32
  3. 2023CoTFormer: A Chain-of-Thought Driven Architecture with Budget-Adaptive Computation Cost at Inferencecited by 27
  4. 2022Optimization with Access to Auxiliary Informationcited by 13
  5. 2024SpaFL: Communication-Efficient Federated Learning with Sparse Models and Low computational Overheadcited by 9
  6. 2023Adaptive Compression in Federated Learning via Side Informationcited by 6
  7. 2023Towards a Better Theoretical Understanding of Independent Subnetwork Trainingcited by 6
  8. 2025Aequa: Fair Model Rewards in Collaborative Learning via Slimmable Networkscited by 6
  9. 2024EncCluster: Scalable Functional Encryption in Federated Learning through Weight Clustering and Probabilistic Filterscited by 3
  10. 2023Straggler-resilient Federated Learning: Tackling Computation Heterogeneity with Layer-wise Partial Model Training in Mobile Edge Networkcited by 2
  11. 2023Federated Fine-Tuning of Foundation Models via Probabilistic Maskingcited by 2
  12. 2025Dropout Neural Network Training Viewed from a Percolation Perspectivecited by 2
4 more
  1. 2023Federated Learning over Hierarchical Wireless Networks: Training Latency Minimization via Submodel Partitioningcited by 1
  2. 2025On the Convergence and Stability of Distributed Sub-model Trainingcited by 1
  3. 2024Delayed Random Partial Gradient Averaging for Federated Learningcited by 0
  4. 2026UAV-Assisted Cooperative Edge Inference for Low-Altitude Economy via MoE-based Hierarchical Deep Reinforcement Learningcited by 0