# Distributed Optimization with Arbitrary Local Solvers

arXiv: https://arxiv.org/abs/1512.04039 · HTML: https://mlsanity.com/paper/1512.04039/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Dec 2015.

## Built on by (most cited first, 23 of 23)

- [Survey on Federated Learning Threats: concepts, taxonomy on attacks and defences, experimental study and challenges](https://mlsanity.com/paper/2201.08135/) (2022)
- [Distributed Second Order Methods with Fast Rates and Compressed Communication](https://mlsanity.com/paper/2102.07158/) (2021)
- [Can 5th Generation Local Training Methods Support Client Sampling? Yes!](https://mlsanity.com/paper/2212.14370/) (2022)
- [Communication Acceleration of Local Gradient Methods via an Accelerated Primal-Dual Algorithm with Inexact Prox](https://mlsanity.com/paper/2207.03957/) (2022)
- [Asynchronous Federated Learning for Sensor Data with Concept Drift](https://mlsanity.com/paper/2109.00151/) (2021)
- [Filling the Missing: Exploring Generative AI for Enhanced Federated Learning over Heterogeneous Mobile Edge Devices](https://mlsanity.com/paper/2310.13981/) (2023)
- [Communication-Efficient Hierarchical Federated Learning for IoT Heterogeneous Systems with Imbalanced Data](https://mlsanity.com/paper/2107.06548/) (2021)
- [Distributed Conditional Generative Adversarial Networks (GANs) for Data-Driven Millimeter Wave Communications in UAV Networks](https://mlsanity.com/paper/2102.01751/) (2021)
- [Adaptive Learning Rates for Faster Stochastic Gradient Methods](https://mlsanity.com/paper/2208.05287/) (2022)
- [Towards a Better Theoretical Understanding of Independent Subnetwork Training](https://mlsanity.com/paper/2306.16484/) (2023)
- [Linear Regression with Distributed Learning: A Generalization Error Perspective](https://mlsanity.com/paper/2101.09001/) (2021)
- [Asynchronous Multi-Model Dynamic Federated Learning over Wireless Networks: Theory, Modeling, and Optimization](https://mlsanity.com/paper/2305.13503/) (2023)
- [FLECS-CGD: A Federated Learning Second-Order Framework via Compression and Sketching with Compressed Gradient Differences](https://mlsanity.com/paper/2210.09626/) (2022)
- [Communication-Efficient, 2D Parallel Stochastic Gradient Descent for Distributed-Memory Optimization](https://mlsanity.com/paper/2501.07526/) (2025)
- [Optimal Privacy Preserving for Federated Learning in Mobile Edge Computing](https://mlsanity.com/paper/2211.07166/) (2022)
- [Continual Learning with Distributed Optimization: Does CoCoA Forget?](https://mlsanity.com/paper/2211.16994/) (2022)
- [Learner Referral for Cost-Effective Federated Learning Over Hierarchical IoT Networks](https://mlsanity.com/paper/2307.09977/) (2023)
- [Optimization of User Selection and Bandwidth Allocation for Federated Learning in VLC/RF Systems](https://mlsanity.com/paper/2103.03444/) (2021)
- [Federated Learning with Superquantile Aggregation for Heterogeneous Data](https://mlsanity.com/paper/2112.09429/) (2021)
- [Client Orchestration and Cost-Efficient Joint Optimization for NOMA-Enabled Hierarchical Federated Learning](https://mlsanity.com/paper/2311.02130/) (2023)
- [Distributed Continual Learning with CoCoA in High-dimensional Linear Regression](https://mlsanity.com/paper/2312.01795/) (2023)
- [Accelerated Training of Federated Learning via Second-Order Methods](https://mlsanity.com/paper/2505.23588/) (2025)
- [Byzantine-Robust Optimization under (L0, L1)-Smoothness](https://mlsanity.com/paper/2603.12512/) (2026)

## Abstract (arXiv)

With the growth of data and necessity for distributed optimization methods, solvers that work well on a single machine must be re-designed to leverage distributed computation. Recent work in this area has been limited by focusing heavily on developing highly specific methods for the distributed environment. These special-purpose methods are often unable to fully leverage the competitive performance of their well-tuned and customized single machine counterparts. Further, they are unable to easily integrate improvements that continue to be made to single machine methods. To this end, we present a framework for distributed optimization that both allows the flexibility of arbitrary solvers to be used on each (single) machine locally, and yet maintains competitive performance against other state-of-the-art special-purpose distributed methods. We give strong primal-dual convergence rate guarantees for our framework that hold for arbitrary local solvers. We demonstrate the impact of local solver selection both theoretically and in an extensive experimental comparison. Finally, we provide thorough implementation details for our framework, highlighting areas for practical performance gains.