# Hogwild! over Distributed Local Data Sets with Linearly Increasing Mini-Batch Sizes

arXiv: https://arxiv.org/abs/2010.14763 · HTML: https://mlsanity.com/paper/2010.14763/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Oct 2020.

## Built on by (most cited first, 5 of 5)

- [Considerations on the Theory of Training Models with Differential Privacy](https://mlsanity.com/paper/2303.04676/) (2023)
- [Proactive DP: A Multple Target Optimization Framework for DP-SGD](https://mlsanity.com/paper/2102.09030/) (2021)
- [AET-SGD: Asynchronous Event-triggered Stochastic Gradient Descent](https://mlsanity.com/paper/2112.13935/) (2021)
- [Generalizing DP-SGD with Shuffling and Batch Clipping](https://mlsanity.com/paper/2212.05796/) (2022)
- [Distributed Learning and its Application for Time-Series Prediction](https://mlsanity.com/paper/2106.03211/) (2021)

## Abstract (arXiv)

Hogwild! implements asynchronous Stochastic Gradient Descent (SGD) where multiple threads in parallel access a common repository containing training data, perform SGD iterations and update shared state that represents a jointly learned (global) model. We consider big data analysis where training data is distributed among local data sets in a heterogeneous way -- and we wish to move SGD computations to local compute nodes where local data resides. The results of these local SGD computations are aggregated by a central "aggregator" which mimics Hogwild!. We show how local compute nodes can start choosing small mini-batch sizes which increase to larger ones in order to reduce communication cost (round interaction with the aggregator). We improve state-of-the-art literature and show O(√K) communication rounds for heterogeneous data for strongly convex problems, where K is the total number of gradient computations across all local compute nodes. For our scheme, we prove a textittight and novel non-trivial convergence analysis for strongly convex problems for em heterogeneous data which does not use the bounded gradient assumption as seen in many existing publications. The tightness is a consequence of our proofs for lower and upper bounds of the convergence rate, which show a constant factor difference. We show experimental results for plain convex and non-convex problems for biased (i.e., heterogeneous) and unbiased local data sets.