# Distributed Coordinate Descent Method for Learning with Big Data

arXiv: https://arxiv.org/abs/1310.2059 · HTML: https://mlsanity.com/paper/1310.2059/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Oct 2013.

## Built on by (most cited first, 23 of 23)

- [A Field Guide to Federated Optimization](https://mlsanity.com/paper/2107.06917/) (2021)
- [Cross-Gradient Aggregation for Decentralized Learning from Non-IID data](https://mlsanity.com/paper/2103.02051/) (2021)
- [Accelerating Heterogeneous Federated Learning with Closed-form Classifiers](https://mlsanity.com/paper/2406.01116/) (2024)
- [FRUGAL: Memory-Efficient Optimization by Reducing State Overhead for Scalable Training](https://mlsanity.com/paper/2411.07837/) (2024)
- [Federated Empirical Risk Minimization via Second-Order Method](https://mlsanity.com/paper/2305.17482/) (2023)
- [Leveraging Coordinate Momentum in SignSGD and Muon: Memory-Optimized Zero-Order](https://mlsanity.com/paper/2506.04430/) (2025)
- [FedBCD:Communication-Efficient Accelerated Block Coordinate Gradient Descent for Federated Learning](https://mlsanity.com/paper/2603.05116/) (2026)
- [Cross-Silo Federated Learning for Multi-Tier Networks with Vertical and Horizontal Data Partitioning](https://mlsanity.com/paper/2108.08930/) (2021)
- [Towards a Better Theoretical Understanding of Independent Subnetwork Training](https://mlsanity.com/paper/2306.16484/) (2023)
- [Stability and Generalization for Randomized Coordinate Descent](https://mlsanity.com/paper/2108.07414/) (2021)
- [Non-Euclidean Broximal Point Method: A Blueprint for Geometry-Aware Optimization](https://mlsanity.com/paper/2510.00823/) (2025)
- [Learning Scalable Model Soup on a Single GPU: An Efficient Subspace Training Strategy](https://mlsanity.com/paper/2407.03641/) (2024)
- [High-Dimensional Distributed Sparse Classification with Scalable Communication-Efficient Global Updates](https://mlsanity.com/paper/2407.06346/) (2024)
- [FLECS-CGD: A Federated Learning Second-Order Framework via Compression and Sketching with Compressed Gradient Differences](https://mlsanity.com/paper/2210.09626/) (2022)
- [A Mirror Descent-Based Algorithm for Corruption-Tolerant Distributed Gradient Descent](https://mlsanity.com/paper/2407.14111/) (2024)
- [Assisted Learning for Organizations with Limited Imbalanced Data](https://mlsanity.com/paper/2109.09307/) (2021)
- [Optimizing the Optimal Weighted Average: Efficient Distributed Sparse Classification](https://mlsanity.com/paper/2406.01753/) (2024)
- [Federated Learning for Commercial Image Sources](https://mlsanity.com/paper/2507.12903/) (2025)
- [Laplacian-based Semi-Supervised Learning in Multilayer Hypergraphs by Coordinate Descent](https://mlsanity.com/paper/2301.12184/) (2023)
- [Batches Stabilize the Minimum Norm Risk in High Dimensional Overparameterized Linear Regression](https://mlsanity.com/paper/2306.08432/) (2023)
- [Scalable High-Dimensional Multivariate Linear Regression for Feature-Distributed Data](https://mlsanity.com/paper/2307.03410/) (2023)
- [Estimation Network Design framework for efficient distributed optimization](https://mlsanity.com/paper/2404.15273/) (2024)
- [Beyond adaptive gradient: Fast-Controlled Minibatch Algorithm for large-scale optimization](https://mlsanity.com/paper/2411.15795/) (2024)

## Abstract (arXiv)

In this paper we develop and analyze Hydra: HYbriD cooRdinAte descent method for solving loss minimization problems with big data. We initially partition the coordinates (features) and assign each partition to a different node of a cluster. At every iteration, each node picks a random subset of the coordinates from those it owns, independently from the other computers, and in parallel computes and applies updates to the selected coordinates based on a simple closed-form formula. We give bounds on the number of iterations sufficient to approximately solve the problem with high probability, and show how it depends on the data and on the partitioning. We perform numerical experiments with a LASSO instance described by a 3TB matrix.