mlsanityData to 7 Oct 2026

MURANA: A Generic Framework for Stochastic Variance-Reduced Optimization

arXiv 2106.03056 ↗First posted 6 Jun 2021math.OC cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

We propose a generic variance-reduced algorithm, which we call MUltiple RANdomized Algorithm (MURANA), for minimizing a sum of several smooth functions plus a regularizer, in a sequential or distributed manner. Our method is formulated with general stochastic operators, which allow us to model various strategies for reducing the computational complexity. For example, MURANA supports sparse activation of the gradients, and also reduction of the communication load via compression of the update vectors. This versatility allows MURANA to cover many existing randomization mechanisms within a unified framework, which also makes it possible to design new methods as special cases.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 33 of its 50 references that are in the map. First posted on arXiv 6 Jun 2021.

Builds on

33 of 50 references.

References · 33

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2019Advances and Open Problems in Federated Learningcited by 3,021
  3. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  4. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  5. 2014Convex Optimization: Algorithms and Complexitycited by 569
  6. 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 415
  7. 2014SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectivescited by 401
  8. 2017TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learningcited by 253
  9. 2019SGD: General Analysis and Improved Ratescited by 206
  10. 2017Gradient Sparsification for Communication-Efficient Distributed Optimizationcited by 199
  11. 2019Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computationscited by 179
  12. 2019Distributed Learning with Compressed Gradient Differencescited by 110
21 more
  1. 2019DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compressioncited by 98
  2. 2019A Unified Theory of SGD: Variance Reduction, Sampling, Quantization and Coordinate Descentcited by 84
  3. 2020Variance-Reduced Methods for Machine Learningcited by 79
  4. 2014A Proximal Stochastic Gradient Method with Progressive Variance Reductioncited by 78
  5. 2019Don't Jump Through Hoops and Remove Those Loops: SVRG and Katyusha are Better Without the Outer Loopcited by 68
  6. 2019Stochastic Distributed Learning with Gradient Quantization and Variance Reductioncited by 64
  7. 2020Linearly Converging Error Compensated SGDcited by 61
  8. 2020Bidirectional compression in heterogeneous settings for distributed or federated learning with partial participation: tight convergence guaranteescited by 45
  9. 2015Variance Reduced Stochastic Gradient Descent with Neighborscited by 39
  10. 2020Optimal Gradient Compression for Distributed and Federated Learningcited by 37
  11. 2020Unified Analysis of Stochastic Gradient Methods for Composite Convex and Smooth Optimizationcited by 36
  12. 2018Stochastic Quasi-Gradient Methods: Variance Reduction via Jacobian Sketchingcited by 34
  13. 2019A Double Residual Compression Algorithm for Efficient Distributed Learningcited by 31
  14. 2019On the Discrepancy between the Theoretical Analysis and Practical Implementations of Compressed Communication for Distributed Deep Learningcited by 31
  15. 2020Fixed Point Strategies in Data Sciencecited by 16
  16. 2019Proximal Splitting Algorithms for Convex Optimization: A Tour of Recent Advances, with New Twistscited by 15
  17. 2019L-SVRG and L-Katyusha with Arbitrary Samplingcited by 14
  18. 2019Optimal mini-batch and step sizes for SAGAcited by 8
  19. 2020Distributed Proximal Splitting Algorithms with Rates and Accelerationcited by 7
  20. 2019Towards closing the gap between the theory and practice of SVRGcited by 6
  21. 2019MISO is Making a Comeback With Better Proofs and Ratescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 5; 2023: 3; 2024: 1; 2025: 1; 2026: 1.

13 papers.

Sort:

  1. 2022Can 5th Generation Local Training Methods Support Client Sampling? Yes!cited by 22
  2. 2022EF-BV: A Unified Theory of Error Feedback and Variance Reduction Mechanisms for Biased and Unbiased Compression in Distributed Optimizationcited by 17
  3. 2021Faster Rates for Compressed Federated Learning with Client-Variance Reductioncited by 14
  4. 2022CompressedScaffnew: The First Theoretical Double Acceleration of Communication from Local Training and Compression in Distributed Optimizationcited by 11
  5. 2024LoCoDL: Communication-Efficient Distributed Learning with Local Training and Compressioncited by 11
  6. 2022QUIC-FL: Quick Unbiased Compression for Federated Learningcited by 7
  7. 2023Improving Accelerated Federated Learning with Compression and Importance Samplingcited by 7
  8. 2022Shifted Compression Framework: Generalizations and Improvementscited by 6
  9. 2023TAMUNA: Doubly Accelerated Distributed Optimization under Partial Participationcited by 3
  10. 2021A Stochastic Variance-Reduced Coordinate Descent Algorithm for Learning Sparse Bayesian Network from Discrete High-Dimensional Datacited by 1
  11. 2026BiCoLoR: Communication-Efficient Optimization with Bidirectional Compression and Local Trainingcited by 1
  12. 2023Unified analysis of SGD-type methodscited by 0
1 more
  1. 2025Gradient Projection onto Historical Descent Directions for Communication-Efficient Federated Learningcited by 0