mlsanityData to 7 Oct 2026

Inverse Online Learning: Understanding Non-Stationary and Reactionary Policies

arXiv 2203.07338 ↗First posted 14 Mar 2022cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Human decision making is well known to be imperfect and the ability to analyse such processes individually is crucial when attempting to aid or improve a decision-maker's ability to perform a task, e.g. to alert them to potential biases or oversights on their part. To do so, it is necessary to develop interpretable representations of how agents make decisions and how this process changes over time as the agent learns online in reaction to the accrued experience. To then understand the decision-making processes underlying a set of observed trajectories, we cast the policy inference problem as the inverse to this online learning problem. By interpreting actions within a potential outcomes framework, we introduce a meaningful mapping based on agents choosing an action they believe to have the greatest treatment effect. We introduce a practical algorithm for retrospectively estimating such perceived effects, alongside the process through which agents update them, using a novel architecture built upon an expressive family of deep state-space models. Through application to the analysis of UNOS organ donation acceptance decisions, we demonstrate that our approach can bring valuable insights into the factors that govern decision processes and how they change over time.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 11 of its 53 references that are in the map. First posted on arXiv 14 Mar 2022.

Builds on

11 of 53 references.

References · 11

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2016Generative Adversarial Imitation Learningcited by 1,576
  3. 2016Variational Inference: A Review for Statisticianscited by 1,466
  4. 2017Meta-learners for Estimating Heterogeneous Treatment Effects using Machine Learningcited by 475
  5. 2018Online Learning: A Comprehensive Surveycited by 203
  6. 2014Taming the Monster: A Fast and Simple Algorithm for Contextual Banditscited by 200
  7. 2016Structured Inference Networks for Nonlinear State Space Modelscited by 182
  8. 2021Scalable Bayesian Inverse Reinforcement Learningcited by 56
  9. 2016Personalized Risk Scoring for Critical Care Prognosis using Mixtures of Gaussian Processescited by 19
  10. 2021Inverse Contextual Bandits: Learning How Behavior Evolves over Timecited by 12
  11. 2021The Medkit-Learn(ing) Environment: Medical Decision Modelling through Simulationcited by 11

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 2; 2024: 1.

6 papers.

  1. 2024Dense Reward for Free in Reinforcement Learning from Human Feedbackcited by 77
  2. 2022POETREE: Interpretable Policy Learning with Adaptive Decision Treescited by 14
  3. 2022Practical Approaches for Fair Learning with Multitype and Multivariate Sensitive Attributescited by 3
  4. 2023When is Off-Policy Evaluation (Reward Modeling) Useful in Contextual Bandits? A Data-Centric Perspectivecited by 3
  5. 2022Synthetic Model Combination: An Instance-wise Approach to Unsupervised Ensemble Learningcited by 2
  6. 2023Optimising Human-AI Collaboration by Learning Convincing Explanationscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2020Learning "What-if" Explanations for Sequential Decision-Making
  2. 2019Modeling and Interpreting Real-world Human Risk Decision Making with Inverse Reinforcement Learning
  3. 2026Ada-Diffuser: Latent-Aware Adaptive Diffusion for Decision-Making
  4. 2026A Lecture Note on Offline RL and IRL, Part II: Foundations of Inverse Reinforcement Learning and Dynamic Discrete Choice Models
  5. 2019On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inference
  6. 2021Uncertain Decisions Facilitate Better Preference Learning
  7. 2026DIML: Differentiable Inverse Mechanism Learning from Behaviors of Multi-Agent Learning Trajectories
  8. 2022Towards Interpretable Deep Reinforcement Learning Models via Inverse Reinforcement Learning