mlsanityData to 7 Oct 2026

Contextual Bilevel Reinforcement Learning for Incentive Alignment

arXiv 2406.01575 ↗First posted 3 Jun 2024math.OC cs.AI cs.LGcited by 14

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

The optimal policy in various real-world strategic decision-making problems depends both on the environmental configuration and exogenous events. For these settings, we introduce Contextual Bilevel Reinforcement Learning (CB-RL), a stochastic bilevel decision-making model, where the lower level consists of solving a contextual Markov Decision Process (CMDP). CB-RL can be viewed as a Stackelberg Game where the leader and a random context beyond the leader's control together decide the setup of many MDPs that potentially multiple followers best respond to. This framework extends beyond traditional bilevel optimization and finds relevance in diverse fields such as RLHF, tax design, reward shaping, contract theory and mechanism design. We propose a stochastic Hyper Policy Gradient Descent (HPGD) algorithm to solve CB-RL, and demonstrate its convergence. Notably, HPGD uses stochastic hypergradient estimates, based on observations of the followers' trajectories. Therefore, it allows followers to use any training procedure and the leader to be agnostic of the specific algorithm, which aligns with various real-world scenarios. We further consider the setting when the leader can influence the training of followers and propose an accelerated algorithm. We empirically demonstrate the performance of our algorithm for reward shaping and tax design.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 7 of its 82 references that are in the map. First posted on arXiv 3 Jun 2024.

Builds on

7 of 82 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Principled Penalty-based Methods for Bilevel Reinforcement Learning and RLHF10 Feb

Other references · 6

  1. 2018Approximation Methods for Bilevel Programmingcited by 231
  2. 2015Contextual Markov Decision Processescited by 213
  3. 2023A Tutorial on Meta-Reinforcement Learningcited by 102
  4. 2024Sample-Efficient Robust Multi-Agent Reinforcement Learning in the Face of Environmental Uncertaintycited by 24
  5. 2024On the Complexity of First-Order Methods in Stochastic Bilevel Optimizationcited by 11
  6. 2021Solving Heterogeneous General Equilibrium Economic Models with Deep Reinforcement Learningcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 9.

14 papers.

Sort:

By area:

  1. 2025Game Theory Meets Large Language Models: A Systematic Survey with Taxonomy and New Frontierscited by 14
  2. 2025Stackelberg Learning from Human Feedback: Preference Optimization as a Sequential Gamecited by 9
  3. 2025Adaptive Context Length Optimization with Low-Frequency Truncation for Multi-Agent Reinforcement Learningcited by 6
  4. 2026Towards Efficient Constraint Handling in Neural Solvers for Routing Problemscited by 2
  5. 2026Sample-Efficient Hypergradient Estimation for Decentralized Bi-Level Reinforcement Learningcited by 2
  6. 2025Zero-Shot Context Generalization in Reinforcement Learning from Few Training Contextscited by 0
  7. 2025DyBBT: Dynamic Balance via Bandit-inspired Targeting for Dialog Policy with Cognitive Dual-Systemscited by 0
  8. 2026DIML: Differentiable Inverse Mechanism Learning from Behaviors of Multi-Agent Learning Trajectoriescited by 0
  9. 2026Sample Complexity Analysis for Constrained Bilevel Reinforcement Learningcited by 0
  10. 2026Deep Incentive Design with Differentiable Equilibrium Blockscited by 0
  11. 2026Evolutionary Bilevel Reward Shaping for Generalization in Reinforcement Learningcited by 0
  12. 2026Hypergradient-based Bilevel Reinforcement Learning with Improved Sample Complexitycited by 0
2 more
  1. 2026Active Inference as Context Acquisition for AI Agentscited by 0
  2. 2026Multi-LLM Collaborative Alignment via Stackelberg Gamescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedback
  2. 2023Stackelberg Batch Policy Learning
  3. 2025Strategyproof Reinforcement Learning from Human Feedback
  4. 2022Strategic Decision-Making in the Presence of Information Asymmetry: Provably Efficient RL with Algorithmic Instruments
  5. 2023BiERL: A Meta Evolutionary Reinforcement Learning Framework via Bilevel Optimization
  6. 2024Direct Preference Optimization for Primitive-Enabled Hierarchical RL: A Bilevel Approach
  7. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  8. 2023Efficient Model-Based Concave Utility Reinforcement Learning through Greedy Mirror Descent