# Divide-and-conquer methods for big data analysis

arXiv: https://arxiv.org/abs/2102.10771 · HTML: https://mlsanity.com/paper/2102.10771/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 48 references that are in the map.
- First posted on arXiv 22 Feb 2021.

## Builds on (most cited first, 13 of 13)

- [Confidence Intervals and Hypothesis Testing for High-Dimensional Regression](https://mlsanity.com/paper/1306.3171/) (2013)
- [Divide and Conquer Kernel Ridge Regression: A Distributed Algorithm with Minimax Optimal Rates](https://mlsanity.com/paper/1305.5029/) (2013)
- [Scalable Bayes via Barycenter in Wasserstein Space](https://mlsanity.com/paper/1508.05880/) (2015)
- [Efficient Distributed Learning with Sparsity](https://mlsanity.com/paper/1605.07991/) (2016)
- [Communication-Efficient Accurate Statistical Estimation](https://mlsanity.com/paper/1906.04870/) (2019)
- [Quantile Regression Under Memory Constraint](https://mlsanity.com/paper/1810.08264/) (2018)
- [Distributed Statistical Estimation and Rates of Convergence in Normal Approximation](https://mlsanity.com/paper/1704.02658/) (2017)
- [A Distributed One-Step Estimator](https://mlsanity.com/paper/1511.01443/) (2015)
- [Robust and Scalable Bayes via a Median of Subset Posterior Measures](https://mlsanity.com/paper/1403.2660/) (2014)
- [Divide and Conquer in Non-standard Problems and the Super-efficiency Phenomenon](https://mlsanity.com/paper/1605.04446/) (2016)
- [A Divide-and-Conquer Bayesian Approach to Large-Scale Kriging](https://mlsanity.com/paper/1712.09767/) (2017)
- [Distributed Simultaneous Inference in Generalized Linear Models via Confidence Distribution](https://mlsanity.com/paper/1611.06208/) (2016)
- [Additive Partially Linear Models for Massive Heterogeneous Data](https://mlsanity.com/paper/1701.03772/) (2017)

## Abstract (arXiv)

In the context of big data analysis, the divide-and-conquer methodology refers to a multiple-step process: first splitting a data set into several smaller ones; then analyzing each set separately; finally combining results from each analysis together. This approach is effective in handling large data sets that are unsuitable to be analyzed entirely by a single computer due to limits either from memory storage or computational time. The combined results will provide a statistical inference which is similar to the one from analyzing the entire data set. This article reviews some recently developments of divide-and-conquer methods in a variety of settings, including combining based on parametric, semiparametric and nonparametric models, online sequential updating methods, among others. Theoretical development on the efficiency of the divide-and-conquer methods is also discussed.