mlsanityData to 14 Oct 2026

Bilevel reinforcement learning via the development of hyper-gradient without lower-level convexity

arXiv 2405.19697 ↗First posted 30 May 2024math.OC cs.AI cs.LGcited by 10

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Bilevel reinforcement learning (RL), which features intertwined two-level problems, has attracted growing interest recently. The inherent non-convexity of the lower-level RL problem is, however, to be an impediment to developing bilevel optimization methods. By employing the fixed point equation associated with the regularized RL, we characterize the hyper-gradient via fully first-order information, thus circumventing the assumption of lower-level convexity. This, remarkably, distinguishes our development of hyper-gradient from the general AID-based bilevel frameworks since we take advantage of the specific structure of RL problems. Moreover, we design both model-based and model-free bilevel reinforcement learning algorithms, facilitated by access to the fully first-order hyper-gradient. Both algorithms enjoy the convergence rate O(ε-1). To extend the applicability, a stochastic version of the model-free algorithm is proposed, along with results on its iteration and sample complexity. In addition, numerical experiments demonstrate that the hyper-gradient indeed serves as an integration of exploitation and exploration.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 68 of its 85 references that are in the map. First posted on arXiv 30 May 2024.

Builds on

68 of 85 references.

References · 68

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2017Deep reinforcement learning from human preferencescited by 3,912
  4. 2015Continuous control with deep reinforcement learningcited by 3,407
  5. 2015Trust Region Policy Optimizationcited by 2,580
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  8. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  9. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  10. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  11. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  12. 2018Bilevel Programming for Hyperparameter Optimization and Meta-Learningcited by 405
38 more
  1. 2018Meta-learning with differentiable closed-form solverscited by 397
  2. 2022Learning robust perceptive locomotion for quadrupedal robots in the wildcited by 382
  3. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 302
  4. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 298
  5. 2019A Theory of Regularized Markov Decision Processescited by 286
  6. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
  7. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 260
  8. 2018Approximation Methods for Bilevel Programmingcited by 236
  9. 2017Forward and Reverse Gradient-Based Hyperparameter Optimizationcited by 234
  10. 2016Hyperparameter optimization with approximate gradientcited by 233
  11. 2020Bilevel Optimization: Convergence Analysis and Enhanced Designcited by 222
  12. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 182
  13. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 177
  14. 2018Understanding the impact of entropy on policy optimizationcited by 176
  15. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 176
  16. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 156
  17. 2020On the Iteration Complexity of Hypergradient Computationcited by 155
  18. 2021Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyondcited by 153
  19. 2020A Two-Timescale Framework for Bilevel Optimization: Complexity Analysis and Application to Actor-Criticcited by 144
  20. 2019Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policiescited by 124
  21. 2020Learning to Utilize Shaping Rewards: A New Approach of Reward Shapingcited by 114
  22. 2021A Near-Optimal Algorithm for Stochastic Bilevel Optimization via Double-Momentumcited by 106
  23. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 106
  24. 2018On Learning Intrinsic Rewards for Policy Gradient Methodscited by 102
  25. 2023A Fully First-Order Method for Stochastic Bilevel Optimizationcited by 100
  26. 2022A framework for bilevel optimization that enables stochastic and global variance reduction algorithmscited by 99
  27. 2023On Penalty-based Bilevel Gradient Descent Methodcited by 87
  28. 2021Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergencecited by 84
  29. 2020A Generic First-Order Algorithmic Framework for Bi-Level Programming Beyond Lower-Level Singletoncited by 79
  30. 2021A Single-Timescale Method for Stochastic Bilevel Optimizationcited by 69
  31. 2021Towards Gradient-based Bilevel Optimization with Non-convex Followers and Beyondcited by 63
  32. 2021Amortized Implicit Differentiation for Stochastic Bilevel Optimizationcited by 63
  33. 2021A Value-Function-based Interior-point Method for Non-convex Bi-level Optimizationcited by 60
  34. 2021A Fully Single Loop Algorithm for Bilevel Optimization without Hessian Inversecited by 53
  35. 2024Principled Penalty-based Methods for Bilevel Reinforcement Learning and RLHFcited by 41
  36. 2023PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedbackcited by 40
  37. 2022Will Bilevel Optimizers Benefit from Loopscited by 36
  38. 2021Enhanced Bilevel Optimization via Bregman Distancecited by 28

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 7.

10 papers.

  1. 2025On The Sample Complexity Bounds In Bilevel Reinforcement Learningcited by 10
  2. 2024BiSSL: Enhancing the Alignment Between Self-Supervised Pretraining and Downstream Fine-Tuning via Bilevel Optimizationcited by 3
  3. 2026Sample-Efficient Hypergradient Estimation for Decentralized Bi-Level Reinforcement Learningcited by 2
  4. 2026Penalty-Based First-Order Methods for Bilevel Optimization with Minimax and Constrained Lower-Level Problemscited by 1
  5. 2026On the Convergence of Self-Improving Online LLM Alignmentcited by 1
  6. 2025From Data to Rewards: a Bilevel Optimization Perspective on Maximum Likelihood Estimationcited by 0
  7. 2026Sample Complexity Analysis for Constrained Bilevel Reinforcement Learningcited by 0
  8. 2026Hypergradient-based Bilevel Reinforcement Learning with Improved Sample Complexitycited by 0
  9. 2026Efficient Hypergradient Descent for Inverse Reinforcement Learningcited by 0
  10. 2026Learning the Robustness Mechanism with Bilevel Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Bilevel Optimization over Saddle Points of Zero-Sum Markov Games
  2. 2024Contextual Bilevel Reinforcement Learning for Incentive Alignment
  3. 2026Non-Stationary Functional Bilevel Optimization
  4. 2026On the Stability and Generalization of First-order Bilevel Minimax Optimization
  5. 2024A Communication and Computation Efficient Fully First-order Method for Decentralized Bilevel Optimization
  6. 2026A Hessian-Free Actor-Critic Algorithm for Bi-Level Reinforcement Learning with Applications to LLM Fine-Tuning
  7. 2024Meta-Reinforcement Learning with Universal Policy Adaptation: Provable Near-Optimality under All-task Optimum Comparator
  8. 2025Bi-Level Policy Optimization with Nystr"om Hypergradients