mlsanityData to 7 Oct 2026

Taming the Wild: A Unified Analysis of Hogwild!-Style Algorithms

arXiv 1506.06438 ↗First posted 22 Jun 2015cs.LG math.OC stat.MLcited by 31

Not on the map. Why?

Abstract (arXiv)

Stochastic gradient descent (SGD) is a ubiquitous algorithm for a variety of machine learning problems. Researchers and industry have developed several techniques to optimize SGD's runtime performance, including asynchronous execution and reduced precision. Our main result is a martingale-based analysis that enables us to capture the rich noise models that may arise from such techniques. Specifically, we use our new analysis in three ways: (1) we derive convergence rates for the convex case (Hogwild!) with relaxed assumptions on the sparsity of the problem; (2) we analyze asynchronous SGD algorithms for non-convex matrix problems including matrix completion; and (3) we design and analyze an asynchronous SGD algorithm, called Buckwild!, that uses lower-precision arithmetic. We show experimentally that our algorithms run efficiently for a variety of problems on modern hardware.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Jun 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 11; 2022: 5; 2023: 7; 2024: 2; 2025: 2; 2026: 3.

31 papers.

Sort:

By area:

  1. 2021Pufferfish: Communication-efficient Models At No Extra Costcited by 61
  2. 2021DataLens: Scalable Privacy Preserving Training via Gradient Compression and Aggregationcited by 31
  3. 2021An optical neural network using less than 1 photon per multiplicationcited by 30
  4. 2021Distributed Learning Systems with First-order Methodscited by 17
  5. 2022A General Theory for Federated Optimization with Asynchronous and Heterogeneous Clients Updatescited by 17
  6. 2021Critical Parameters for Scalable Distributed Learning with Large Batches and Asynchronous Updatescited by 16
  7. 2023Stochastic Approximation Beyond Gradient for Signal Processing and Machine Learningcited by 10
  8. 2023Spectrum Breathing: Protecting Over-the-Air Federated Learning Against Interferencecited by 9
  9. 2023Convergence and Privacy of Decentralized Nonconvex Optimization with Gradient Clipping and Communication Compressioncited by 9
  10. 2022Transformers Go for the LOLs: Generating (Humourous) Titles from Scientific Abstracts End-to-Endcited by 6
  11. 2020Elastic Consistency: A General Consistency Model for Distributed Stochastic Gradient Descentcited by 3
  12. 2025Low-Precision Streaming PCAcited by 3
19 more
  1. 2023Considerations on the Theory of Training Models with Differential Privacycited by 2
  2. 2023Distributed Extra-gradient with Optimal Complexity and Communication Guaranteescited by 2
  3. 2021Proactive DP: A Multple Target Optimization Framework for DP-SGDcited by 1
  4. 2021AET-SGD: Asynchronous Event-triggered Stochastic Gradient Descentcited by 1
  5. 2022Scaling the Wild: Decentralizing Hogwild!-style Shared-memory SGDcited by 1
  6. 2022Scaling up Stochastic Gradient Descent for Non-convex Optimisationcited by 1
  7. 2022Quantization-Based Optimization: Alternative Stochastic Approximation of Global Optimizationcited by 1
  8. 2024Dual-Delayed Asynchronous SGD for Arbitrarily Heterogeneous Datacited by 1
  9. 2024Acoustic Model Optimization over Multiple Data Sources: Merging and Valuationcited by 1
  10. 2025DP-CSGP: Differentially Private Stochastic Gradient Push with Compressed Communicationcited by 1
  11. 2026Tight Analysis of Decentralized SGD: A Markov Chain Perspectivecited by 1
  12. 2021NUQSGD: Provably Communication-efficient Data-parallel SGD via Nonuniform Quantizationcited by 0
  13. 2021Scalable Projection-Free Optimizationcited by 0
  14. 2021Distributed Learning and its Application for Time-Series Predictioncited by 0
  15. 2021Accelerating Perturbed Stochastic Iterates in Asynchronous Lock-Free Optimizationcited by 0
  16. 2023Stochastic Gradient Langevin Dynamics Based on Quantization with Increasing Resolutioncited by 0
  17. 2023The Landscape of Modern Machine Learning: A Review of Machine, Distributed and Federated Learningcited by 0
  18. 2026Asynchronous Heavy-Tailed Optimizationcited by 0
  19. 2026Revisiting Gradient Staleness: Evaluating Distance Metrics for Asynchronous Federated Learning Aggregationcited by 0