# Inverse Contextual Bandits: Learning How Behavior Evolves over Time

arXiv: https://arxiv.org/abs/2107.06317 · HTML: https://mlsanity.com/paper/2107.06317/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 81 references that are in the map.
- First posted on arXiv 13 Jul 2021.

## Builds on (most cited first, 28 of 28)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Counterfactual Visual Explanations](https://mlsanity.com/paper/1904.07451/) (2019)
- [Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations](https://mlsanity.com/paper/1904.06387/) (2019)
- [What do we need to build explainable AI systems for the medical domain?](https://mlsanity.com/paper/1712.09923/) (2017)
- [Robust Imitation of Diverse Behaviors](https://mlsanity.com/paper/1707.02747/) (2017)
- [Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences](https://mlsanity.com/paper/2002.09089/) (2020)
- [Online Bayesian Goal Inference for Boundedly-Rational Planning Agents](https://mlsanity.com/paper/2006.07532/) (2020)
- [TACO: Learning Task Decomposition via Temporal Alignment for Control](https://mlsanity.com/paper/1803.01840/) (2018)
- [Strictly Batch Imitation Learning by Energy-based Distribution Matching](https://mlsanity.com/paper/2006.14154/) (2020)
- [Directed-Info GAIL: Learning Hierarchical Policies from Unsegmented Demonstrations using Directed Information](https://mlsanity.com/paper/1810.01266/) (2018)
- [Where Do You Think You're Going?: Inferring Beliefs about Dynamics from Behavior](https://mlsanity.com/paper/1805.08010/) (2018)
- [Adversarial Imitation via Variational Inverse Reinforcement Learning](https://mlsanity.com/paper/1809.06404/) (2018)
- [Model-based Adversarial Imitation Learning](https://mlsanity.com/paper/1612.02179/) (2016)
- [Deep Bayesian Reward Learning from Preferences](https://mlsanity.com/paper/1912.04472/) (2019)
- [Truly Batch Apprenticeship Learning with Deep Successor Features](https://mlsanity.com/paper/1903.10077/) (2019)
- [f-GAIL: Learning f-Divergence for Generative Adversarial Imitation Learning](https://mlsanity.com/paper/2010.01207/) (2020)
- [What Did You Think Would Happen? Explaining Agent Behaviour Through Intended Outcomes](https://mlsanity.com/paper/2011.05064/) (2020)
- [The Assistive Multi-Armed Bandit](https://mlsanity.com/paper/1901.08654/) (2019)
- [Efficient Exploration of Reward Functions in Inverse Reinforcement Learning via Bayesian Optimization](https://mlsanity.com/paper/2011.08541/) (2020)
- [Inverse Active Sensing: Modeling and Understanding Timely Decision-Making](https://mlsanity.com/paper/2006.14141/) (2020)
- [On-Policy Robot Imitation Learning from a Converging Supervisor](https://mlsanity.com/paper/1907.03423/) (2019)
- [Learning a Multi-Modal Policy via Imitating Demonstrations with Mixed Behaviors](https://mlsanity.com/paper/1903.10304/) (2019)
- [Learning from an Exploring Demonstrator: Optimal Reward Estimation for Bandits](https://mlsanity.com/paper/2106.14866/) (2021)
- [Inverse Reinforcement Learning from a Gradient-based Learner](https://mlsanity.com/paper/2007.07812/) (2020)
- [Modelling Agent Policies with Interpretable Imitation Learning](https://mlsanity.com/paper/2006.11309/) (2020)
- [Am I Building a White Box Agent or Interpreting a Black Box Agent?](https://mlsanity.com/paper/2007.01187/) (2020)

## Built on by (most cited first, 12 of 12)

- [Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examples](https://mlsanity.com/paper/2310.07747/) (2023)
- [Inverse Online Learning: Understanding Non-Stationary and Reactionary Policies](https://mlsanity.com/paper/2203.07338/) (2022)
- [Online Decision Mediation](https://mlsanity.com/paper/2310.18601/) (2023)
- [When is Off-Policy Evaluation (Reward Modeling) Useful in Contextual Bandits? A Data-Centric Perspective](https://mlsanity.com/paper/2311.14110/) (2023)
- [Defining Expertise: Applications to Treatment Effect Estimation](https://mlsanity.com/paper/2403.00694/) (2024)
- [Optimising Human-AI Collaboration by Learning Convincing Explanations](https://mlsanity.com/paper/2311.07426/) (2023)
- [IBCB: Efficient Inverse Batched Contextual Bandit for Behavioral Evolution History](https://mlsanity.com/paper/2403.16075/) (2024)
- [Efficient and Interpretable Bandit Algorithms](https://mlsanity.com/paper/2310.14751/) (2023)
- [Fitting Reinforcement Learning Model to Behavioral Data under Bandits](https://mlsanity.com/paper/2511.04454/) (2025)
- [Inverse Contextual Bandits without Rewards: Learning from a Non-Stationary Learner via Suffix Imitation](https://mlsanity.com/paper/2603.03778/) (2026)
- [OTSS: Output-Targeted Soft Segmentation for Contextual Decision-Weight Learning](https://mlsanity.com/paper/2605.00193/) (2026)
- [Quantifying Potential Observation Missingness in Inverse Reinforcement Learning](https://mlsanity.com/paper/2605.12831/) (2026)

## Abstract (arXiv)

Understanding a decision-maker's priorities by observing their behavior is critical for transparency and accountability in decision processes, such as in healthcare. Though conventional approaches to policy learning almost invariably assume stationarity in behavior, this is hardly true in practice: Medical practice is constantly evolving as clinical professionals fine-tune their knowledge over time. For instance, as the medical community's understanding of organ transplantations has progressed over the years, a pertinent question is: How have actual organ allocation policies been evolving? To give an answer, we desire a policy learning method that provides interpretable representations of decision-making, in particular capturing an agent's non-stationary knowledge of the world, as well as operating in an offline manner. First, we model the evolving behavior of decision-makers in terms of contextual bandits, and formalize the problem of Inverse Contextual Bandits (ICB). Second, we propose two concrete algorithms as solutions, learning parametric and nonparametric representations of an agent's behavior. Finally, using both real and simulated data for liver transplantations, we illustrate the applicability and explainability of our method, as well as benchmarking and validating its accuracy.