mlsanityData to 7 Oct 2026

FeDXL: Provable Federated Learning for Deep X-Risk Optimization

arXiv 2210.14396 ↗First posted 26 Oct 2022cs.LG cs.DC math.OCcited by 10

Not on the map. Why?

Abstract (arXiv)

In this paper, we tackle a novel federated learning (FL) problem for optimizing a family of X-risks, to which no existing FL algorithms are applicable. In particular, the objective has the form of mathbb Ez~ S1 f(mathbb Ez'~ S2 ℓ(w; z, z')), where two sets of data S1, S2 are distributed over multiple machines, ℓ(·) is a pairwise loss that only depends on the prediction outputs of the input data pairs (z, z'), and f(·) is possibly a non-linear non-convex function. This problem has important applications in machine learning, e.g., AUROC maximization with a pairwise loss, and partial AUROC maximization with a compositional loss. The challenges for designing an FL algorithm for X-risks lie in the non-decomposability of the objective over multiple machines and the interdependency between different machines. To this end, we propose an active-passive decomposition framework that decouples the gradient's components with two types, namely active parts and passive parts, where the active parts depend on local data that are computed with the local model and the passive parts depend on other machines that are communicated/computed based on historical models and samples. Under this framework, we develop two provable FL algorithms (FeDXL) for handling linear and nonlinear f, respectively, based on federated averaging and merging. We develop a novel theoretical analysis to combat the latency of the passive parts and the interdependency between the local model parameters and the involved data for computing local gradient estimators. We establish both iteration and communication complexities and show that using the historical samples and models for computing the passive parts do not degrade the complexities. We conduct empirical studies of FeDXL for deep AUROC and partial AUROC maximization, and demonstrate their performance compared with several baselines.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 48 of its 71 references that are in the map. First posted on arXiv 26 Oct 2022.

Builds on

48 of 71 references.

References · 48

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2019Advances and Open Problems in Federated Learningcited by 3,021
  4. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  5. 2021MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classificationcited by 682
  6. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  7. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
  8. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 558
  9. 2018Sparsified SGD with Memorycited by 373
  10. 2017Sampling Matters in Deep Embedding Learningcited by 285
  11. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 277
  12. 2019Tighter Theory for Local SGD on Identical and Heterogeneous Datacited by 266
36 more
  1. 2017Gradient Sparsification for Communication-Efficient Distributed Optimizationcited by 199
  2. 2019Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computationscited by 179
  3. 2019On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimizationcited by 178
  4. 2020Mime: Mimicking Centralized Stochastic Algorithms in Federated Learningcited by 152
  5. 2020Is Local SGD Better than Minibatch SGD?cited by 151
  6. 2020Minibatch vs Local SGD for Heterogeneous Distributed Learningcited by 147
  7. 2019Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronizationcited by 106
  8. 2021Distributionally Robust Federated Averagingcited by 104
  9. 2022AUC Maximization in the Era of Big Data and AI: A Surveycited by 100
  10. 2014Stochastic Compositional Gradient Descent: Algorithms for Minimizing Compositions of Expected-Value Functionscited by 93
  11. 2020Federated Unsupervised Representation Learningcited by 92
  12. 2020Large-scale Robust Deep AUC Maximization: A New Surrogate Loss and Empirical Studies on Medical Image Classificationcited by 84
  13. 2016CoCoA: A General Framework for Communication-Efficient Distributed Optimizationcited by 69
  14. 2022FedNest: Federated Bilevel, Minimax, and Compositional Optimizationcited by 60
  15. 2016CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examplescited by 58
  16. 2018A Single Time-Scale Stochastic Approximation Method for Nested Stochastic Optimizationcited by 53
  17. 2019On Symmetric Losses for Learning from Corrupted Labelscited by 53
  18. 2020Biased Stochastic First-Order Methods for Conditional Stochastic Optimization and Applications in Meta Learningcited by 46
  19. 2019A Decentralized Parallel Algorithm for Training Generative Adversarial Netscited by 43
  20. 2013One-Pass AUC Optimizationcited by 42
  21. 2021Local Stochastic Gradient Descent Ascent: Convergence Analysis and Communication Efficiencycited by 40
  22. 2022Provable Stochastic Optimization for Global Contrastive Learning: Small Batch Does Not Harm Performancecited by 40
  23. 2022FedX: Unsupervised Federated Learning with Cross Knowledge Distillationcited by 40
  24. 2022Federated Minimax Optimization: Improved Convergence Analyses and Algorithmscited by 37
  25. 2021Stochastic Optimization of Areas Under Precision-Recall Curves with Provable Convergencecited by 34
  26. 2022When AUC meets DRO: Optimizing Partial AUC for Deep Learning with Non-Convex Convergence Guaranteecited by 29
  27. 2020Communication-Efficient Distributed Stochastic AUC Maximization with Deep Neural Networkscited by 27
  28. 2022Finite-Sum Coupled Compositional Stochastic Optimization: Theory and Applicationscited by 27
  29. 2021Federated Deep AUC Maximization for Heterogeneous Data with a Constant Communication Complexitycited by 21
  30. 2022Large-scale Stochastic Optimization of NDCG Surrogates for Deep Learning with Provable Convergencecited by 21
  31. 2021Simple Stochastic and Online Gradient Descent Algorithms for Pairwise Learningcited by 20
  32. 2022Multi-block-Single-probe Variance Reduced Estimator for Coupled Compositional Optimizationcited by 16
  33. 2021Momentum Accelerates the Convergence of Stochastic AUPRC Maximizationcited by 14
  34. 2022Communication-Efficient Robust Federated Learning with Noisy Labelscited by 11
  35. 2022Federated Learning on Adaptively Weighted Nodes by Bilevel Optimizationcited by 6
  36. 2022FedGRec: Federated Graph Recommender System with Lazy Update of Latent Embeddingscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 2; 2025: 2; 2026: 2.

10 papers.

  1. 2023Solving a Class of Non-Convex Minimax Optimization in Federated Learningcited by 10
  2. 2023Federated Compositional Deep AUC Maximizationcited by 8
  3. 2025Frequency-Semantic Enhanced Variational Autoencoder for Zero-Shot Skeleton-based Action Recognitioncited by 6
  4. 2023Federated Conditional Stochastic Optimizationcited by 4
  5. 2023Serverless Federated AUPRC Optimization for Multi-Party Collaborative Imbalanced Data Miningcited by 3
  6. 2024Communication-Efficient Federated Group Distributionally Robust Optimizationcited by 3
  7. 2024FedNE: Surrogate-Assisted Federated Neighbor Embedding for Dimensionality Reductioncited by 1
  8. 2025Closing the Approximation Gap of Partial AUC Optimization: A Tale of Two Formulationscited by 0
  9. 2026Communication-Efficient Federated AUC Maximization with Cyclic Client Participationcited by 0
  10. 2026Provably Communication-Efficient and Privacy-Preserving Federated Graph Neural Networkscited by 0