mlsanityData to 14 Oct 2026

SimFBO: Towards Simple, Flexible and Communication-efficient Federated Bilevel Learning

arXiv 2305.19442 ↗First posted 30 May 2023cs.LG cs.DC math.OCcited by 14

Not on the map. Why?

Abstract (arXiv)

Federated bilevel optimization (FBO) has shown great potential recently in machine learning and edge computing due to the emerging nested optimization structure in meta-learning, fine-tuning, hyperparameter tuning, etc. However, existing FBO algorithms often involve complicated computations and require multiple sub-loops per iteration, each of which contains a number of communication rounds. In this paper, we propose a simple and flexible FBO framework named SimFBO, which is easy to implement without sub-loops, and includes a generalized server-side aggregation and update for improving communication efficiency. We further propose System-level heterogeneity robust FBO (ShroFBO) as a variant of SimFBO with stronger resilience to heterogeneous local computation. We show that SimFBO and ShroFBO provably achieve a linear convergence speedup with partial client participation and client sampling without replacement, as well as improved sample and communication complexities. Experiments demonstrate the effectiveness of the proposed methods over existing FBO algorithms.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 54 of its 78 references that are in the map. First posted on arXiv 30 May 2023.

Builds on

54 of 78 references.

References · 54

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  3. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  4. 2020Adaptive Federated Optimizationcited by 1,046
  5. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  6. 2018Local SGD Converges Fast and Communicates Littlecited by 605
  7. 2019Meta-Learning with Implicit Gradientscited by 525
  8. 2019Agnostic Federated Learningcited by 456
  9. 2018Bilevel Programming for Hyperparameter Optimization and Meta-Learningcited by 405
  10. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 360
  11. 2020Personalized Federated Learning: A Meta-Learning Approachcited by 312
  12. 2018Approximation Methods for Bilevel Programmingcited by 236
38 more
  1. 2017Forward and Reverse Gradient-Based Hyperparameter Optimizationcited by 234
  2. 2016Hyperparameter optimization with approximate gradientcited by 233
  3. 2020Bilevel Optimization: Convergence Analysis and Enhanced Designcited by 222
  4. 2015Federated Optimization:Distributed Optimization Beyond the Datacentercited by 195
  5. 2018Truncated Back-propagation for Bilevel Optimizationcited by 190
  6. 2019Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computationscited by 181
  7. 2020On the Iteration Complexity of Hypergradient Computationcited by 155
  8. 2021Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyondcited by 153
  9. 2017A Review on Bilevel Optimization: From Classical to Evolutionary Approaches and Applicationscited by 129
  10. 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 123
  11. 2016On Differentiating Parameterized Argmin and Argmax Problems with Application to Bi-level Optimizationcited by 112
  12. 2020FedSplit: An algorithmic framework for fast federated optimizationcited by 108
  13. 2021A Near-Optimal Algorithm for Stochastic Bilevel Optimization via Double-Momentumcited by 106
  14. 2021Provably Faster Algorithms for Bilevel Optimizationcited by 106
  15. 2022BOME! Bilevel Optimization Made Easy: A Simple First-Order Approachcited by 105
  16. 2020FairBatch: Batch Selection for Model Fairnesscited by 99
  17. 2022A framework for bilevel optimization that enables stochastic and global variance reduction algorithmscited by 99
  18. 2023On Penalty-based Bilevel Gradient Descent Methodcited by 87
  19. 2019First Analysis of Local GD on Heterogeneous Datacited by 81
  20. 2020A Generic First-Order Algorithmic Framework for Bi-Level Programming Beyond Lower-Level Singletoncited by 79
  21. 2021Revisiting and Advancing Fast Adversarial Training Through The Lens of Bi-Level Optimizationcited by 76
  22. 2018Reviving and Improving Recurrent Back-Propagationcited by 69
  23. 2017A First Order Method for Solving Convex Bi-Level Optimization Problemscited by 67
  24. 2021Towards Gradient-based Bilevel Optimization with Non-convex Followers and Beyondcited by 63
  25. 2021Amortized Implicit Differentiation for Stochastic Bilevel Optimizationcited by 63
  26. 2021A Value-Function-based Interior-point Method for Non-convex Bi-level Optimizationcited by 60
  27. 2022FedNest: Federated Bilevel, Minimax, and Compositional Optimizationcited by 60
  28. 2021Improving Fairness via Federated Learningcited by 39
  29. 2021Randomized Stochastic Variance-Reduced Methods for Multi-Task Stochastic Bilevel Optimizationcited by 33
  30. 2021BiAdam: Fast Adaptive Bilevel Optimization Methodscited by 33
  31. 2022Decentralized Gossip-Based Stochastic Bilevel Optimization over Communication Networkscited by 32
  32. 2020Improved Bilevel Model: Fast and Optimal Algorithm with Theoretical Guaranteecited by 28
  33. 2022Decentralized Bilevel Optimizationcited by 23
  34. 2022Local Stochastic Bilevel Optimization with Momentum-Based Variance Reductioncited by 20
  35. 2023Achieving Linear Speedup in Non-IID Federated Bilevel Learningcited by 18
  36. 2015Strategies and Principles of Distributed Machine Learning on Big Datacited by 16
  37. 2023Communication-Efficient Federated Hypergradient Computation via Aggregated Iterative Differentiationcited by 13
  38. 2023Federated Minimax Optimization with Client Heterogeneitycited by 8

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 7; 2025: 4; 2026: 2.

14 papers.

Sort:

  1. 2024Constrained Bi-Level Optimization: Proximal Lagrangian Value function Approach and Hessian-free Algorithmcited by 16
  2. 2025Direct Prediction Set Minimization via Bilevel Conformal Classifier Trainingcited by 8
  3. 2023A Single-Loop Algorithm for Decentralized Bilevel Optimizationcited by 7
  4. 2025LDC-MTL: Balancing Multi-Task Learning through Scalable Loss Discrepancy Controlcited by 7
  5. 2024Tuning-Free Bilevel Optimization: New Algorithms and Convergence Analysiscited by 5
  6. 2025Multiplayer Federated Learning: Reaching Equilibrium with Less Communicationcited by 5
  7. 2024Review of Mathematical Optimization in Federated Learningcited by 3
  8. 2024Fully First-Order Methods for Decentralized Bilevel Optimizationcited by 1
  9. 2024FedDTPT: Federated Discrete and Transferable Prompt Tuning for Black-Box Large Language Modelscited by 1
  10. 2024Provably Faster Algorithms for Bilevel Optimization via Without-Replacement Samplingcited by 1
  11. 2024Communication-Efficient and Privacy-Preserving Decentralized Meta-Learningcited by 0
  12. 2025A Randomized Zeroth-Order Hierarchical Framework for Heterogeneous Federated Learningcited by 0
2 more
  1. 2026BROS: Bias-Corrected Randomized Subspaces for Memory-Efficient Single-Loop Bilevel Optimizationcited by 0
  2. 2026Federated stochastic bilevel optimization with fully first-order gradientscited by 0