# Efficient Distributed Learning with Sparsity

arXiv: https://arxiv.org/abs/1605.07991 · HTML: https://mlsanity.com/paper/1605.07991/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 May 2016.

## Built on by (most cited first, 30 of 38)

- [Fine-tuning Language Models over Slow Networks using Activation Compression with Guarantees](https://mlsanity.com/paper/2206.01299/) (2022)
- [An Adaptive Stochastic Sequential Quadratic Programming with Differentiable Exact Augmented Lagrangians](https://mlsanity.com/paper/2102.05320/) (2021)
- [Federated Causal Inference in Heterogeneous Observational Data](https://mlsanity.com/paper/2107.11732/) (2021)
- [Targeting Underrepresented Populations in Precision Medicine: A Federated Transfer Learning Approach](https://mlsanity.com/paper/2108.12112/) (2021)
- [Distributed Learning Systems with First-order Methods](https://mlsanity.com/paper/2104.05245/) (2021)
- [Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameters](https://mlsanity.com/paper/2111.05897/) (2021)
- [BAGUA: Scaling up Distributed Learning with System Relaxations](https://mlsanity.com/paper/2107.01499/) (2021)
- [Fast Composite Optimization and Statistical Recovery in Federated Learning](https://mlsanity.com/paper/2207.08204/) (2022)
- [Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism](https://mlsanity.com/paper/2506.01260/) (2025)
- [A Communication-efficient Algorithm with Linear Convergence for Federated Minimax Learning](https://mlsanity.com/paper/2206.01132/) (2022)
- [Communication-Constrained Distributed Quantile Regression with Optimal Statistical Guarantees](https://mlsanity.com/paper/2110.13113/) (2021)
- [Shift is Good: Mismatched Data Mixing Improves Test Performance](https://mlsanity.com/paper/2510.25108/) (2025)
- [Distributed Bootstrap for Simultaneous Inference Under High Dimensionality](https://mlsanity.com/paper/2102.10080/) (2021)
- [Fast and Robust Sparsity Learning over Networks: A Decentralized Surrogate Median Regression Approach](https://mlsanity.com/paper/2202.05498/) (2022)
- [Sharper Analysis for Minibatch Stochastic Proximal Point Methods: Stability, Smoothness, and Deviation](https://mlsanity.com/paper/2301.03125/) (2023)
- [CG-FedLLM: How to Compress Gradients in Federated Fune-tuning for Large Language Models](https://mlsanity.com/paper/2405.13746/) (2024)
- [Distributed Sparse Normal Means Estimation with Sublinear Communication](https://mlsanity.com/paper/2102.03060/) (2021)
- [High-Dimensional Distributed Sparse Classification with Scalable Communication-Efficient Global Updates](https://mlsanity.com/paper/2407.06346/) (2024)
- [AsyncMesh: Fully Asynchronous Optimization for Data and Pipeline Parallelism](https://mlsanity.com/paper/2601.22442/) (2026)
- [Distributed Sparse Feature Selection in Communication-Restricted Networks](https://mlsanity.com/paper/2111.02802/) (2021)
- [Distributed High-Dimensional Quantile Regression: Estimation Efficiency and Support Recovery](https://mlsanity.com/paper/2405.07552/) (2024)
- [Distributed Sparse Regression via Penalization](https://mlsanity.com/paper/2111.06530/) (2021)
- [CEDAR: Communication Efficient Distributed Analysis for Regressions](https://mlsanity.com/paper/2207.00306/) (2022)
- [Majority Vote for Distributed Differentially Private Sign Selection](https://mlsanity.com/paper/2209.04419/) (2022)
- [Distributed Semi-Supervised Sparse Statistical Inference](https://mlsanity.com/paper/2306.10395/) (2023)
- [Optimizing the Optimal Weighted Average: Efficient Distributed Sparse Classification](https://mlsanity.com/paper/2406.01753/) (2024)
- [Learning Pareto manifolds in high dimensions: How can regularization help?](https://mlsanity.com/paper/2503.08849/) (2025)
- [Learned Subspace Compression for Communication-Efficient Pipeline Parallelism](https://mlsanity.com/paper/2606.05484/) (2026)
- [Divide-and-conquer methods for big data analysis](https://mlsanity.com/paper/2102.10771/) (2021)
- [Communication-efficient Byzantine-robust distributed learning with statistical guarantee](https://mlsanity.com/paper/2103.00373/) (2021)

## Abstract (arXiv)

We propose a novel, efficient approach for distributed sparse learning in high-dimensions, where observations are randomly partitioned across machines. Computationally, at each round our method only requires the master machine to solve a shifted ell_1 regularized M-estimation problem, and other workers to compute the gradient. In respect of communication, the proposed approach provably matches the estimation error bound of centralized methods within constant rounds of communications (ignoring logarithmic factors). We conduct extensive experiments on both simulated and real world datasets, and demonstrate encouraging performances on high-dimensional regression and classification tasks.