mlsanityData to 7 Oct 2026

IBCB: Efficient Inverse Batched Contextual Bandit for Behavioral Evolution History

arXiv 2403.16075 ↗First posted 24 Mar 2024cs.LGcited by 1

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Traditional imitation learning focuses on modeling the behavioral mechanisms of experts, which requires a large amount of interaction history generated by some fixed expert. However, in many streaming applications, such as streaming recommender systems, online decision-makers typically engage in online learning during the decision-making process, meaning that the interaction history generated by online decision-makers includes their behavioral evolution from novice expert to experienced expert. This poses a new challenge for existing imitation learning approaches that can only utilize data from experienced experts. To address this issue, this paper proposes an inverse batched contextual bandit (IBCB) framework that can efficiently perform estimations of environment reward parameters and learned policy based on the expert's behavioral evolution history. Specifically, IBCB formulates the inverse problem into a simple quadratic programming problem by utilizing the behavioral evolution history of the batched contextual bandit with inaccessible rewards. We demonstrate that IBCB is a unified framework for both deterministic and randomized bandit policies. The experimental results indicate that IBCB outperforms several existing imitation learning algorithms on synthetic and real-world data and significantly reduces running time. Additionally, empirical analyses reveal that IBCB exhibits better out-of-distribution generalization and is highly effective in learning the bandit policy from the interaction history of novice experts.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 28 of its 61 references that are in the map. First posted on arXiv 24 Mar 2024.

Builds on

28 of 61 references.

Closest ancestors · 3

Same direction, newest first

  1. 2021Imitation Learning: Progress, Taxonomies and Challenges23 Jun
  2. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learning27 Oct
  3. 2019On-Policy Robot Imitation Learning from a Converging Supervisor8 Jul

Other references · 25

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  3. 2016Generative Adversarial Imitation Learningcited by 1,576
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  6. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  7. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
  8. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 294
  9. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
  10. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
  11. 2017OSQP: An Operator Splitting Solver for Quadratic Programscited by 156
  12. 2020Sequential Batch Learning in Finite-Action Linear Contextual Banditscited by 63
13 more
  1. 2018Online Learning with an Unknown Fairness Metriccited by 61
  2. 2020Impact of Representation Learning in Linear Banditscited by 51
  3. 2021Fairness of Exposure in Stochastic Banditscited by 42
  4. 2018Adversarial Imitation via Variational Inverse Reinforcement Learningcited by 36
  5. 2018Balanced Linear Contextual Banditscited by 30
  6. 2019Deep Bayesian Reward Learning from Preferencescited by 28
  7. 2020Dynamic Batch Learning in High-Dimensional Sparse Linear Contextual Banditscited by 28
  8. 2020An Imitation Learning Approach for Cache Replacementcited by 24
  9. 2021Batched Neural Banditscited by 22
  10. 2016Streaming Recommender Systemscited by 17
  11. 2020Generative Inverse Deep Reinforcement Learning for Online Recommendationcited by 12
  12. 2021Inverse Contextual Bandits: Learning How Behavior Evolves over Timecited by 12
  13. 2022Exposure-Aware Recommendation using Contextual Banditscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Inverse Contextual Bandits without Rewards: Learning from a Non-Stationary Learner via Suffix Imitationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Contextual Bandits and Imitation Learning via Preference-Based Active Queries
  2. 2023A Strong Baseline for Batch Imitation Learning
  3. 2019Meta-Learning for Contextual Bandit Exploration
  4. 2024Inverse Reinforcement Learning by Estimating Expertise of Demonstrators
  5. 2019A Divergence Minimization Perspective on Imitation Learning Methods
  6. 2022A Ranking Game for Imitation Learning
  7. 2022On Efficient Online Imitation Learning via Classification
  8. 2024Reward-free World Models for Online Imitation Learning