mlsanityData to 14 Oct 2026

Trustworthiness of Stochastic Gradient Descent in Distributed Learning

arXiv 2410.21491 ↗First posted 28 Oct 2024cs.LG cs.AI cs.DCcited by 2

Not on the map. Why?

Abstract (arXiv)

Distributed learning (DL) uses multiple nodes to accelerate training, enabling efficient optimization of large-scale models. Stochastic Gradient Descent (SGD), a key optimization algorithm, plays a central role in this process. However, communication bottlenecks often limit scalability and efficiency, leading to increasing adoption of compressed SGD techniques to alleviate these challenges. Despite addressing communication overheads, compressed SGD introduces trustworthiness concerns, as gradient exchanges among nodes are vulnerable to attacks like gradient inversion (GradInv) and membership inference attacks (MIA). The trustworthiness of compressed SGD remains unexplored, leaving important questions about its reliability unanswered. In this paper, we provide a trustworthiness evaluation of compressed versus uncompressed SGD. Specifically, we conducted empirical studies using GradInv attacks, revealing that compressed SGD demonstrates significantly higher resistance to privacy leakage compared to uncompressed SGD. In addition, our findings suggest that MIA may not be a reliable metric for assessing privacy risks in distributed learning.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 22 of its 29 references that are in the map. First posted on arXiv 28 Oct 2024.

Builds on

22 of 29 references.

References · 22

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  4. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  5. 2019Deep Leakage from Gradientscited by 1,021
  6. 2020Inverting Gradients -- How easy is it to break privacy in federated learning?cited by 712
  7. 2018Exploiting Unintended Feature Leakage in Collaborative Learningcited by 520
  8. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 478
  9. 2018Sparsified SGD with Memorycited by 376
  10. 2021See through Gradients: Image Batch Recovery via GradInversioncited by 326
  11. 2019Error Feedback Fixes SignSGD and other Gradient Compression Schemescited by 310
  12. 2020Systematic Evaluation of Privacy Risks of Machine Learning Modelscited by 292
10 more
  1. 2017Sparse Communication for Distributed Gradient Descentcited by 249
  2. 2019PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimizationcited by 238
  3. 2018The Convergence of Sparsified Gradient Methodscited by 229
  4. 2017Gradient Sparsification for Communication-Efficient Distributed Optimizationcited by 201
  5. 2021Evaluating Gradient Inversion Attacks and Defenses in Federated Learningcited by 196
  6. 2020On Biased Compression for Distributed Learningcited by 140
  7. 2018ATOMO: Communication-efficient Learning via Atomic Sparsificationcited by 136
  8. 2020A Framework for Evaluating Gradient Leakage Attacks in Federated Learningcited by 64
  9. 2024Evaluations of Machine Learning Privacy Defenses are Misleadingcited by 36
  10. 2022On the Privacy Effect of Data Enhancement via the Lens of Memorizationcited by 15

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Lightweight Trustworthy Distributed Clusteringcited by 1
  2. 2025Trustworthy Efficient Communication for Distributed Learning using LQ-SGD Algorithmcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025An Improved Privacy and Utility Analysis of Differentially Private SGD with Bounded Domain and Smooth Losses
  2. 2023Killing Two Birds with One Stone: Quantization Achieves Privacy in Distributed Learning
  3. 2022Gradient Obfuscation Gives a False Sense of Security in Federated Learning
  4. 2022sqSGD: Locally Private and Communication Efficient Federated Learning
  5. 2021FastSGD: A Fast Compressed SGD Framework for Distributed Machine Learning
  6. 2025Unbiased and Sign Compression in Distributed Learning: Comparing Noise Resilience via SDEs
  7. 2024Gradient Inversion of Federated Diffusion Models
  8. 2024Flattened one-bit stochastic gradient descent: compressed distributed optimization with controlled variance