mlsanityData to 14 Oct 2026

Interactive Inverse Reinforcement Learning of Interaction Scenarios via Bi-level Optimization

arXiv 2605.08131 ↗First posted 1 May 2026cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

also in 1 more
Abstract (arXiv)

Inverse reinforcement learning (IRL) learns a reward function and a corresponding policy that best fit the demonstration data of an expert. However, in the current IRL setting, the learner is isolated from the expert and can only passively observe the expert demonstrations. This limits the applicability of IRL to interactive settings, where the learner actively interacts with the expert and needs to infer the expert's reward function from the interactions. To bridge the gap, this paper studies interactive IRL (IIRL) where a learner aims to learn the reward function of an expert and a policy to interact with the expert during its interactions with the expert. We formulate IIRL as a stochastic bi-level optimization problem where the lower level learns a reward function to explain the behaviors of the expert, and the upper level learns a policy to interact with the expert. We develop a double-loop algorithm, Bi-level Interactive Scenarios Inverse Reinforcement Learning (BISIRL), which solves the lower-level problem in the inner loop and the upper-level problem in the outer loop. We formally guarantee that BISIRL converges and validate our algorithm through extensive experiments.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 22 of its 47 references that are in the map. First posted on arXiv 1 May 2026.

Builds on

22 of 47 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024In-Trajectory Inverse Reinforcement Learning: Learn Incrementally Before An Ongoing Trajectory Terminates21 Oct
  2. 2024Convergence of a model-free entropy-regularized inverse reinforcement learning algorithm25 Mar
  3. 2023When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learning15 Feb
  4. 2022Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guarantees4 Oct
  5. 2019Non-Cooperative Inverse Reinforcement Learning3 Nov
  6. 2019Multi-Agent Adversarial Inverse Reinforcement Learning30 Jul
  7. 2019Interactive Teaching Algorithms for Inverse Reinforcement Learning28 May
  8. 2018Competitive Multi-agent Inverse Reinforcement Learning with Sub-optimal Demonstrations7 Jan

Other references · 14

  1. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  2. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 942
  3. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  4. 2019The StarCraft Multi-Agent Challengecited by 597
  5. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 485
  6. 2019Meta-Learning with Differentiable Convex Optimizationcited by 461
  7. 2016Cooperative Inverse Reinforcement Learningcited by 300
  8. 2018Approximation Methods for Bilevel Programmingcited by 236
  9. 2016Hyperparameter optimization with approximate gradientcited by 233
  10. 2019Neural Policy Gradient Methods: Global Optimality and Rates of Convergencecited by 159
  11. 2019Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policiescited by 124
  12. 2016First-Person Activity Forecasting with Online Inverse Reinforcement Learningcited by 40
2 more
  1. 2023Efficient Gradient Approximation Method for Constrained Bilevel Optimizationcited by 15
  2. 2021Interactive Inverse Reinforcement Learning for Cooperative Gamescited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Active Exploration for Inverse Reinforcement Learning
  2. 2025BiCQL-ML: A Bi-Level Conservative Q-Learning Framework for Maximum Likelihood Inverse Reinforcement Learning
  3. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress
  4. 2023Data-Driven Inverse Reinforcement Learning for Expert-Learner Zero-Sum Games
  5. 2024Inverse Reinforcement Learning with Sub-optimal Experts
  6. 2025Inverse Reinforcement Learning via Convex Optimization
  7. 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective
  8. 2023BC-IRL: Learning Generalizable Reward Functions from Demonstrations