# Inverse Online Learning: Understanding Non-Stationary and Reactionary Policies

arXiv: https://arxiv.org/abs/2203.07338 · HTML: https://mlsanity.com/paper/2203.07338/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 53 references that are in the map.
- First posted on arXiv 14 Mar 2022.

## Builds on (most cited first, 11 of 11)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Meta-learners for Estimating Heterogeneous Treatment Effects using Machine Learning](https://mlsanity.com/paper/1706.03461/) (2017)
- [Online Learning: A Comprehensive Survey](https://mlsanity.com/paper/1802.02871/) (2018)
- [Taming the Monster: A Fast and Simple Algorithm for Contextual Bandits](https://mlsanity.com/paper/1402.0555/) (2014)
- [Structured Inference Networks for Nonlinear State Space Models](https://mlsanity.com/paper/1609.09869/) (2016)
- [Scalable Bayesian Inverse Reinforcement Learning](https://mlsanity.com/paper/2102.06483/) (2021)
- [Personalized Risk Scoring for Critical Care Prognosis using Mixtures of Gaussian Processes](https://mlsanity.com/paper/1610.08853/) (2016)
- [Inverse Contextual Bandits: Learning How Behavior Evolves over Time](https://mlsanity.com/paper/2107.06317/) (2021)
- [The Medkit-Learn(ing) Environment: Medical Decision Modelling through Simulation](https://mlsanity.com/paper/2106.04240/) (2021)

## Built on by (most cited first, 6 of 6)

- [Dense Reward for Free in Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2402.00782/) (2024)
- [POETREE: Interpretable Policy Learning with Adaptive Decision Trees](https://mlsanity.com/paper/2203.08057/) (2022)
- [When is Off-Policy Evaluation (Reward Modeling) Useful in Contextual Bandits? A Data-Centric Perspective](https://mlsanity.com/paper/2311.14110/) (2023)
- [Practical Approaches for Fair Learning with Multitype and Multivariate Sensitive Attributes](https://mlsanity.com/paper/2211.06138/) (2022)
- [Synthetic Model Combination: An Instance-wise Approach to Unsupervised Ensemble Learning](https://mlsanity.com/paper/2210.05320/) (2022)
- [Optimising Human-AI Collaboration by Learning Convincing Explanations](https://mlsanity.com/paper/2311.07426/) (2023)

## Abstract (arXiv)

Human decision making is well known to be imperfect and the ability to analyse such processes individually is crucial when attempting to aid or improve a decision-maker's ability to perform a task, e.g. to alert them to potential biases or oversights on their part. To do so, it is necessary to develop interpretable representations of how agents make decisions and how this process changes over time as the agent learns online in reaction to the accrued experience. To then understand the decision-making processes underlying a set of observed trajectories, we cast the policy inference problem as the inverse to this online learning problem. By interpreting actions within a potential outcomes framework, we introduce a meaningful mapping based on agents choosing an action they believe to have the greatest treatment effect. We introduce a practical algorithm for retrospectively estimating such perceived effects, alongside the process through which agents update them, using a novel architecture built upon an expressive family of deep state-space models. Through application to the analysis of UNOS organ donation acceptance decisions, we demonstrate that our approach can bring valuable insights into the factors that govern decision processes and how they change over time.