mlsanityData to 7 Oct 2026

Hypergradient-based Bilevel Reinforcement Learning with Improved Sample Complexity

arXiv 2607.28849 ↗First posted 30 Jul 2026cs.LG cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Bilevel reinforcement learning (RL) is an important framework within the literature of RL that can be used to formalize various categories of problems, such as meta-learning, hierarchical task decomposition, and reinforcement learning from human feedback (RL-HF). Most of the bilevel RL algorithms are either not scalable because of using hypergradient with Hessian, or they suffer from high sample complexity because of using penalty-based approximation methods. In this work, we propose a hypergradient-based bilevel RL algorithm using the optimality of the Boltzmann policy for the entropy regularized discounted RL objective function. Our proposed algorithm is Hessian-free and obtains an iteration complexity of O(ε-1) and state-of-the-art sample complexity of tildeO(ε-2) under mild regularity conditions. Further, in our convergence analysis, we are able to remove the assumption of the Polyak-Lojasiewicz (PL) condition on the outer-level objective function present in the prior state-of-the-art sample complexity work.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 28 of its 34 references that are in the map. First posted on arXiv 30 Jul 2026.

Builds on

28 of 34 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Contextual Bilevel Reinforcement Learning for Incentive Alignment3 Jun
  2. 2024Principled Penalty-based Methods for Bilevel Reinforcement Learning and RLHF10 Feb
  3. 2023PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedback3 Aug
  4. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 24

  1. 2018DeepMind Control Suitecited by 935
  2. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 303
  3. 2018Approximation Methods for Bilevel Programmingcited by 231
  4. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 178
  5. 2019Neural Policy Gradient Methods: Global Optimality and Rates of Convergencecited by 157
  6. 2020A Two-Timescale Framework for Bilevel Optimization: Complexity Analysis and Application to Actor-Criticcited by 141
  7. 2021A Near-Optimal Algorithm for Stochastic Bilevel Optimization via Double-Momentumcited by 106
  8. 2021Provably Faster Algorithms for Bilevel Optimizationcited by 106
  9. 2019Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functionscited by 104
  10. 2022BOME! Bilevel Optimization Made Easy: A Simple First-Order Approachcited by 103
  11. 2023A Fully First-Order Method for Stochastic Bilevel Optimizationcited by 97
  12. 2023On Penalty-based Bilevel Gradient Descent Methodcited by 87
12 more
  1. 2021A Single-Timescale Method for Stochastic Bilevel Optimizationcited by 69
  2. 2022An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methodscited by 63
  3. 2021A Value-Function-based Interior-point Method for Non-convex Bi-level Optimizationcited by 60
  4. 2021A Fully Single Loop Algorithm for Bilevel Optimization without Hessian Inversecited by 53
  5. 2023On Penalty Methods for Nonconvex Bilevel Optimization and First-Order Stochastic Approximationcited by 50
  6. 2023On Finding Small Hyper-Gradients in Bilevel Optimization: Hardness Results and Improved Analysiscited by 39
  7. 2020Single-Timescale Actor-Critic Provably Finds Globally Optimal Policycited by 34
  8. 2021On the Convergence Theory for Hessian-Free Bilevel Algorithmscited by 33
  9. 2022Finite-time analysis of single-timescale actor-criticcited by 30
  10. 2022Bilevel Optimization with a Lower-level Contraction: Optimal Sample Complexity without Warm-startcited by 17
  11. 2024Bilevel reinforcement learning via the development of hyper-gradient without lower-level convexitycited by 10
  12. 2025On The Sample Complexity Bounds In Bilevel Reinforcement Learningcited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026A Hessian-Free Actor-Critic Algorithm for Bi-Level Reinforcement Learning with Applications to LLM Fine-Tuning
  2. 2026Efficient Hypergradient Descent for Inverse Reinforcement Learning
  3. 2023BiERL: A Meta Evolutionary Reinforcement Learning Framework via Bilevel Optimization
  4. 2024Direct Preference Optimization for Primitive-Enabled Hierarchical RL: A Bilevel Approach
  5. 2024Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization
  6. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons
  7. 2024Towards Sample-Efficiency and Generalization of Transfer and Inverse Reinforcement Learning: A Comprehensive Literature Review
  8. 2025One Framework to Rule Them All: Unifying RL-Based and RL-Free Methods in RLHF