mlsanityData to 14 Oct 2026

A Selective Review on Statistical Methods for Massive Data Computation: Distributed Computing, Subsampling, and Minibatch Techniques

arXiv 2403.11163 ↗First posted 17 Mar 2024stat.ME cs.LG math.STcited by 0

Not on the map. Why?

Abstract (arXiv)

This paper presents a selective review of statistical computation methods for massive data analysis. A huge amount of statistical methods for massive data computation have been rapidly developed in the past decades. In this work, we focus on three categories of statistical computation methods: (1) distributed computing, (2) subsampling methods, and (3) minibatch gradient techniques. The first class of literature is about distributed computing and focuses on the situation, where the dataset size is too huge to be comfortably handled by one single computer. In this case, a distributed computation system with multiple computers has to be utilized. The second class of literature is about subsampling methods and concerns about the situation, where the sample size of dataset is small enough to be placed on one single computer but too large to be easily processed by its memory as a whole. The last class of literature studies those minibatch gradient related optimization techniques, which have been extensively used for optimizing various deep learning models.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 92 of its 190 references that are in the map. First posted on arXiv 17 Mar 2024.

Builds on

92 of 190 references.

References · 92

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  6. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  7. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  8. 2016An overview of gradient descent optimization algorithmscited by 1,033
  9. 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 664
  10. 2018Local SGD Converges Fast and Communicates Littlecited by 605
  11. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 502
  12. 2014SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectivescited by 402
38 more
  1. 2019SGD: General Analysis and Improved Ratescited by 207
  2. 2013On the Convergence of Decentralized Gradient Descentcited by 159
  3. 2016Achieving Geometric Convergence for Distributed Optimization over Time-Varying Graphscited by 150
  4. 2013Communication Efficient Distributed Optimization using an Approximate Newton-type Methodcited by 132
  5. 2020An Improved Analysis of Stochastic Gradient Descent with Momentumcited by 119
  6. 2017Online Learning Rate Adaptation with Hypergradient Descentcited by 118
  7. 2020Random Reshuffling: Simple Analysis with Vast Improvementscited by 115
  8. 2013Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)cited by 111
  9. 2017Bridging the Gap between Constant Step Size Stochastic Gradient Descent and Markov Chainscited by 88
  10. 2020A Blockchain-based Decentralized Federated Learning Framework with Committee Consensuscited by 84
  11. 2015Why Random Reshuffling Beats Stochastic Gradient Descentcited by 79
  12. 2019Federated Learning with Cooperating Devices: A Consensus Approach for Massive IoT Networkscited by 79
  13. 2016Decentralized Collaborative Learning of Personalized Models over Networkscited by 76
  14. 2016Statistical Inference for Model Parameters in Stochastic Gradient Descentcited by 71
  15. 2019The Step Decay Schedule: A Near Optimal, Geometrically Decaying Learning Rate Procedure For Least Squarescited by 68
  16. 2020On Linear Stochastic Approximation: Fine-grained Polyak-Ruppert and Non-Asymptotic Concentrationcited by 68
  17. 2017Personalized and Private Peer-to-Peer Machine Learningcited by 54
  18. 2017Variants of RMSProp and Adagrad with Logarithmic Regret Boundscited by 50
  19. 2017Momentum and Stochastic Momentum for Stochastic Gradient, Newton, Proximal Point and Subspace Descent Methodscited by 49
  20. 2018Quasi-hyperbolic momentum and Adam for deep learningcited by 48
  21. 2020An Analysis of Constant Step Size SGD in the Non-convex Regime: Asymptotic Normality and Biascited by 48
  22. 2017GIANT: Globally Improved Approximate Newton Method for Distributed Optimizationcited by 44
  23. 2019Understanding the Role of Momentum in Stochastic Gradient Methodscited by 44
  24. 2013A Statistical Perspective on Algorithmic Leveragingcited by 39
  25. 2016Efficient Distributed Learning with Sparsitycited by 38
  26. 2019Augment your batch: better training with larger batchescited by 37
  27. 2020Online Covariance Matrix Estimation in Stochastic Gradient Descentcited by 37
  28. 2019Asynchronous Decentralized SGD with Quantized and Local Updatescited by 36
  29. 2016Gossip training for deep learningcited by 35
  30. 2016Barzilai-Borwein Step Size for Stochastic Gradient Descentcited by 33
  31. 2016Parallel SGD: When does averaging help?cited by 32
  32. 2019Communication-Efficient Accurate Statistical Estimationcited by 32
  33. 2017Optimal Subsampling for Large Sample Logistic Regressioncited by 29
  34. 2020On the Convergence of Nesterov's Accelerated Gradient Method in Stochastic Settingscited by 27
  35. 2017Accelerated Distributed Nesterov Gradient Descentcited by 25
  36. 2019DINGO: Distributed Newton-Type Method for Gradient-Norm Optimizationcited by 20
  37. 2019Faster Neural Network Training with Data Echoingcited by 20
  38. 2020On the Promise of the Stochastic Generalized Gauss-Newton Method for Training DNNscited by 19

50 most cited of 92 All 92 (JSON)

Built on by

Not cited yet.