mlsanityData to 7 Oct 2026

Federated Conditional Stochastic Optimization

arXiv 2310.02524 ↗First posted 4 Oct 2023cs.LG cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Conditional stochastic optimization has found applications in a wide range of machine learning tasks, such as invariant learning, AUPRC maximization, and meta-learning. As the demand for training models with large-scale distributed data grows in these applications, there is an increasing need for communication-efficient distributed optimization algorithms, such as federated learning algorithms. This paper considers the nonconvex conditional stochastic optimization in federated learning and proposes the first federated conditional stochastic optimization algorithm (FCSG) with a conditional stochastic gradient estimator and a momentum-based algorithm (FCSG-M). To match the lower bound complexity in the single-machine setting, we design an accelerated algorithm (Acc-FCSG-M) via the variance reduction to achieve the best sample and communication complexity. Compared with the existing optimization analysis for MAML in FL, federated conditional stochastic optimization considers the sample of tasks. Extensive experimental results on various tasks validate the efficiency of these algorithms.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 33 of its 53 references that are in the map. First posted on arXiv 4 Oct 2023.

Builds on

33 of 53 references.

References · 33

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  3. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  4. 2016Matching Networks for One Shot Learningcited by 2,246
  5. 2020Adaptive Federated Optimizationcited by 1,036
  6. 2020Personalized Federated Learning: A Meta-Learning Approachcited by 309
  7. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 302
  8. 2018SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimatorcited by 283
  9. 2018Federated Meta-Learning with Fast Convergence and Efficient Communicationcited by 154
  10. 2020Mime: Mimicking Centralized Stochastic Algorithms in Federated Learningcited by 152
  11. 2019Kernel Instrumental Variable Regressioncited by 118
  12. 2014Stochastic Compositional Gradient Descent: Algorithms for Minimizing Compositions of Expected-Value Functionscited by 93
21 more
  1. 2020Bridging Composite and Real: Towards End-to-end Deep Image Mattingcited by 89
  2. 2019Torchmeta: A Meta-Learning library for PyTorchcited by 52
  3. 2020Faster Non-Convex Federated Learning via Global and Local Momentumcited by 48
  4. 2016Accelerating Stochastic Composition Optimizationcited by 46
  5. 2020Biased Stochastic First-Order Methods for Conditional Stochastic Optimization and Applications in Meta Learningcited by 46
  6. 2021STEM: A Stochastic Two-Sided Momentum Algorithm Achieving Near-Optimal Sample and Communication Complexities for Federated Learningcited by 43
  7. 2022Faster Adaptive Federated Learningcited by 35
  8. 2021Stochastic Optimization of Areas Under Precision-Recall Curves with Provable Convergencecited by 34
  9. 2020Communication-Efficient Distributed Stochastic AUC Maximization with Deep Neural Networkscited by 27
  10. 2022Finite-Sum Coupled Compositional Stochastic Optimization: Theory and Applicationscited by 27
  11. 2021Federated Deep AUC Maximization for Heterogeneous Data with a Constant Communication Complexitycited by 21
  12. 2023MAC-PO: Multi-Agent Experience Replay via Collective Priority Optimizationcited by 20
  13. 2019Sample Complexity of Sample Average Approximation for Conditional Stochastic Optimizationcited by 19
  14. 2021Momentum Accelerates the Convergence of Stochastic AUPRC Maximizationcited by 14
  15. 2022An Accelerated Doubly Stochastic Gradient Method with Faster Explicit Model Identificationcited by 14
  16. 2015Learning with Group Invariant Features: A Kernel Perspectivecited by 13
  17. 2017Accelerated Method for Stochastic Composition Optimization with Nonsmooth Regularizationcited by 13
  18. 2021Compositional federated learning: Applications in distributionally robust averaging and meta learningcited by 12
  19. 2022FeDXL: Provable Federated Learning for Deep X-Risk Optimizationcited by 10
  20. 2020Stochastic Recursive Momentum Method for Non-Convex Compositional Optimizationcited by 6
  21. 2023Serverless Federated AUPRC Optimization for Multi-Party Collaborative Imbalanced Data Miningcited by 3

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 2.

4 papers.

  1. 2023Every Parameter Matters: Ensuring the Convergence of Federated Learning with Dynamic Heterogeneous Models Reductioncited by 24
  2. 2023Solving a Class of Non-Convex Minimax Optimization in Federated Learningcited by 10
  3. 2024Bayesian Optimization through Gaussian Cox Process Models for Spatio-temporal Datacited by 5
  4. 2024Non-asymptotic Analysis of Biased Adaptive Stochastic Approximationcited by 4