# Adaptive Sampling Distributed Stochastic Variance Reduced Gradient for Heterogeneous Distributed Datasets

arXiv: https://arxiv.org/abs/2002.08528 · HTML: https://mlsanity.com/paper/2002.08528/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Feb 2020.

## Built on by (most cited first, 4 of 4)

- [Lifelong Matrix Completion with Sparsity-Number](https://mlsanity.com/paper/2203.07637/) (2022)
- [Adaptive Noisy Matrix Completion](https://mlsanity.com/paper/2203.08340/) (2022)
- [Performance of Distribution Regression with Doubling Measure under the seek of Closest Point](https://mlsanity.com/paper/2203.00155/) (2022)
- [Matrix Completion with Heterogonous Cost](https://mlsanity.com/paper/2203.12120/) (2022)

## Abstract (arXiv)

We study distributed optimization algorithms for minimizing the average of heterogeneous functions distributed across several machines with a focus on communication efficiency. In such settings, naively using the classical stochastic gradient descent (SGD) or its variants (e.g., SVRG) with a uniform sampling of machines typically yields poor performance. It often leads to the dependence of convergence rate on maximum Lipschitz constant of gradients across the devices. In this paper, we propose a novel adaptive sampling of machines specially catered to these settings. Our method relies on an adaptive estimate of local Lipschitz constants base on the information of past gradients. We show that the new way improves the dependence of convergence rate from maximum Lipschitz constant to average Lipschitz constant across machines, thereby, significantly accelerating the convergence. Our experiments demonstrate that our method indeed speeds up the convergence of the standard SVRG algorithm in heterogeneous environments.