mlsanityData to 14 Oct 2026

BiCQL-ML: A Bi-Level Conservative Q-Learning Framework for Maximum Likelihood Inverse Reinforcement Learning

arXiv 2511.22210 ↗First posted 27 Nov 2025cs.LG cs.ROcited by 0

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

also in 1 more
Abstract (arXiv)

Offline inverse reinforcement learning (IRL) aims to recover a reward function that explains expert behavior using only fixed demonstration data, without any additional online interaction. We propose BiCQL-ML, a policy-free offline IRL algorithm that jointly optimizes a reward function and a conservative Q-function in a bi-level framework, thereby avoiding explicit policy learning. The method alternates between (i) learning a conservative Q-function via Conservative Q-Learning (CQL) under the current reward, and (ii) updating the reward parameters to maximize the expected Q-values of expert actions while suppressing over-generalization to out-of-distribution actions. This procedure can be viewed as maximum likelihood estimation under a soft value matching principle. We provide theoretical guarantees that BiCQL-ML converges to a reward function under which the expert policy is soft-optimal. Empirically, we show on standard offline RL benchmarks that BiCQL-ML improves both reward recovery and downstream policy performance compared to existing offline IRL baselines.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 10 of its 22 references that are in the map. First posted on arXiv 27 Nov 2025.

Builds on

10 of 22 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learning15 Feb

Other references · 9

  1. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  2. 2016Generative Adversarial Imitation Learningcited by 1,597
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  5. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  6. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  7. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
  8. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learningcited by 204
  9. 2019Imitation Learning via Off-Policy Distribution Matchingcited by 183

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Loop-Free Inverse Reinforcement Learning via Sequential Value Recovery with Q-Score Matching
  2. 2024ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
  3. 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective
  4. 2024Bayesian Inverse Reinforcement Learning for Non-Markovian Rewards
  5. 2023A Bayesian Approach to Robust Inverse Reinforcement Learning
  6. 2026Interactive Inverse Reinforcement Learning of Interaction Scenarios via Bi-level Optimization
  7. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning
  8. 2026Q-based Variational Inverse Reinforcement Learning