mlsanityData to 7 Oct 2026

Asynchronous Distributed Bilevel Optimization

arXiv 2212.10048 ↗First posted 20 Dec 2022cs.LG cs.AI math.OCcited by 13

Not on the map. Why?

Abstract (arXiv)

Bilevel optimization plays an essential role in many machine learning tasks, ranging from hyperparameter optimization to meta-learning. Existing studies on bilevel optimization, however, focus on either centralized or synchronous distributed setting. The centralized bilevel optimization approaches require collecting massive amount of data to a single server, which inevitably incur significant communication expenses and may give rise to data privacy risks. Synchronous distributed bilevel optimization algorithms, on the other hand, often face the straggler problem and will immediately stop working if a few workers fail to respond. As a remedy, we propose Asynchronous Distributed Bilevel Optimization (ADBO) algorithm. The proposed ADBO can tackle bilevel optimization problems with both nonconvex upper-level and lower-level objective functions, and its convergence is theoretically guaranteed. Furthermore, it is revealed through theoretic analysis that the iteration complexity of ADBO to obtain the ε-stationary point is upper bounded by O(frac1ε 2). Thorough empirical studies on public datasets have been conducted to elucidate the effectiveness and efficiency of the proposed ADBO.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 37 of its 70 references that are in the map. First posted on arXiv 20 Dec 2022.

Builds on

37 of 70 references.

References · 37

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  3. 2021Convex optimizationcited by 2,479
  4. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  5. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  6. 2019Meta-Learning with Implicit Gradientscited by 523
  7. 2018Bilevel Programming for Hyperparameter Optimization and Meta-Learningcited by 403
  8. 2018Meta-learning with differentiable closed-form solverscited by 395
  9. 2019Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAMLcited by 359
  10. 2018Approximation Methods for Bilevel Programmingcited by 231
  11. 2020Bilevel Optimization: Convergence Analysis and Enhanced Designcited by 219
  12. 2019A Survey on Distributed Machine Learningcited by 187
25 more
  1. 2020On the Iteration Complexity of Hypergradient Computationcited by 154
  2. 2017A Review on Bilevel Optimization: From Classical to Evolutionary Approaches and Applicationscited by 126
  3. 2016On Differentiating Parameterized Argmin and Argmax Problems with Application to Bi-level Optimizationcited by 108
  4. 2021A Near-Optimal Algorithm for Stochastic Bilevel Optimization via Double-Momentumcited by 106
  5. 2021Provably Faster Algorithms for Bilevel Optimizationcited by 106
  6. 2021Revisiting and Advancing Fast Adversarial Training Through The Lens of Bi-Level Optimizationcited by 75
  7. 2018Reviving and Improving Recurrent Back-Propagationcited by 69
  8. 2021A Single-Timescale Method for Stochastic Bilevel Optimizationcited by 69
  9. 2020Convergence of Meta-Learning with Task-Specific Adaptation over Partial Parameterscited by 65
  10. 2021Towards Gradient-based Bilevel Optimization with Non-convex Followers and Beyondcited by 62
  11. 2021A Value-Function-based Interior-point Method for Non-convex Bi-level Optimizationcited by 60
  12. 2022FedNest: Federated Bilevel, Minimax, and Compositional Optimizationcited by 60
  13. 2020A Unified Single-loop Alternating Gradient Projection Algorithm for Nonconvex-Concave and Convex-Nonconcave Minimax Problemscited by 56
  14. 2019Hybrid Block Successive Approximation for One-Sided Non-Convex Min-Max Problems: Algorithms and Applicationscited by 55
  15. 2020Advances in Asynchronous Parallel and Distributed Optimizationcited by 40
  16. 2018Robust Optimization over Multiple Domainscited by 34
  17. 2019Penalty Method for Inversion-Free Deep Bilevel Optimizationcited by 33
  18. 2021Asynchronous Stochastic Optimization Robust to Arbitrary Delayscited by 32
  19. 2022Decentralized Gossip-Based Stochastic Bilevel Optimization over Communication Networkscited by 32
  20. 2022Decentralized Bilevel Optimizationcited by 23
  21. 2022Local Stochastic Bilevel Optimization with Momentum-Based Variance Reductioncited by 20
  22. 2022RoMFAC: A robust mean-field actor-critic reinforcement learning against adversarial perturbations on statescited by 16
  23. 2022Distributed Distributionally Robust Optimization with Non-Convex Objectivescited by 14
  24. 2015Asynchronous Distributed ADMM for Large-Scale Optimization- Part I: Algorithm and Convergence Analysiscited by 10
  25. 2021Debiasing a First-order Heuristic for Approximate Bi-level Optimizationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 2; 2025: 4; 2026: 2.

13 papers.

Sort:

By area:

  1. 2024Tri-Level Navigator: LLM-Empowered Tri-Level Learning for Time Series OOD Generalizationcited by 12
  2. 2023Communication-Efficient Federated Bilevel Optimization with Local and Global Lower Level Problemscited by 8
  3. 2023Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challengescited by 5
  4. 2023Federated Distributionally Robust Optimization with Non-Convex Objectives: Algorithm and Analysiscited by 4
  5. 2025Cellular Traffic Prediction via Byzantine-robust Asynchronous Federated Learningcited by 4
  6. 2024Triadic-OCD: Asynchronous Online Change Detection with Provable Robustness, Optimality, and Convergencecited by 2
  7. 2025Argus: Federated Non-convex Bilevel Learning over 6G Space-Air-Ground Integrated Networkcited by 2
  8. 2023Decentralized and Privacy-Preserving Learning of Approximate Stackelberg Solutions in Energy Trading Games with Demand Response Aggregatorscited by 0
  9. 2023Federated Multi-Sequence Stochastic Approximation with Local Hypergradient Estimationcited by 0
  10. 2025UrbanMind: Towards Urban General Intelligence via Tool-Enhanced Retrieval-Augmented Generation and Multilevel Optimizationcited by 0
  11. 2025Synergistic Localization and Sensing in MIMO-OFDM Systems via Mixed-Integer Bilevel Learningcited by 0
  12. 2026AsynDBT: Asynchronous Distributed Bilevel Tuning for efficient In-Context Learning with Large Language Modelscited by 0
1 more
  1. 2026First-order Constrained Trilevel Optimization Over Distributed Networks for Robust Coreset Selectioncited by 0