mlsanityData to 7 Oct 2026

A Bayesian Solution To The Imitation Gap

arXiv 2407.00495 ↗First posted 29 Jun 2024cs.LGcited by 2

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

In many real-world settings, an agent must learn to act in environments where no reward signal can be specified, but a set of expert demonstrations is available. Imitation learning (IL) is a popular framework for learning policies from such demonstrations. However, in some cases, differences in observability between the expert and the agent can give rise to an imitation gap such that the expert's policy is not optimal for the agent and a naive application of IL can fail catastrophically. In particular, if the expert observes the Markov state and the agent does not, then the expert will not demonstrate the information-gathering behavior needed by the agent but not the expert. In this paper, we propose a Bayesian solution to the Imitation Gap (BIG), first using the expert demonstrations, together with a prior specifying the cost of exploratory behavior that is not demonstrated, to infer a posterior over rewards with Bayesian inverse reinforcement learning (IRL). BIG then uses the reward posterior to learn a Bayes-optimal policy. Our experiments show that BIG, unlike IL, allows the agent to explore at test time when presented with an imitation gap, whilst still learning to behave optimally using expert demonstrations when no such gap exists.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 26 of its 39 references that are in the map. First posted on arXiv 29 Jun 2024.

Builds on

26 of 39 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024A Survey of Imitation Learning Methods, Environments and Metrics30 Apr
  2. 2022Deconfounding Imitation Learning with Variational Inference4 Nov
  3. 2022Sequence Model Imitation Learning with Unobserved Contexts3 Aug
  4. 2021Seeing Differently, Acting Similarly: Heterogeneously Observable Imitation Learning17 Jun
  5. 2020Bridging the Imitation Gap by Adaptive Insubordination23 Jul

Other references · 21

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2013Auto-Encoding Variational Bayescited by 3,980
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2016Generative Adversarial Imitation Learningcited by 1,576
  5. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  6. 2019Learning by Cheatingcited by 361
  7. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
  8. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 294
  9. 2018Provably Efficient Maximum Entropy Explorationcited by 240
  10. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 210
  11. 2023A Tutorial on Meta-Reinforcement Learningcited by 102
  12. 2021Shaking the foundations: delusions in sequence models for interaction and controlcited by 61
9 more
  1. 2018Successor Uncertainties: Exploration and Uncertainty in Temporal Difference Learningcited by 35
  2. 2019Deep Bayesian Reward Learning from Preferencescited by 28
  3. 2020Exploration in Approximate Hyper-State Space for Meta Reinforcement Learningcited by 28
  4. 2021PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learningcited by 23
  5. 2022Leveraging Fully Observable Policies for Learning under Partial Observabilitycited by 17
  6. 2019Inverse Rational Control with Partially Observable Continuous Nonlinear Dynamicscited by 16
  7. 2023Why Target Networks Stabilise Temporal Difference Methodscited by 13
  8. 2023Bayesian Exploration Networkscited by 3
  9. 2023Probabilistic inverse optimal control for non-linear partially observable systems disentangles perceptual uncertainty and behavioral costscited by 1

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024IGDrivSim: A Benchmark for the Imitation Gap in Autonomous Drivingcited by 3
  2. 2025Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatismcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2021Scalable Bayesian Inverse Reinforcement Learning
  2. 2026Nonparametric Bayesian Inverse Reinforcement Learning with Data-Parallel Gibbs Sampling
  3. 2021Imitation Learning by Reinforcement Learning
  4. 2020Bayesian Robust Optimization for Imitation Learning
  5. 2022A Ranking Game for Imitation Learning
  6. 2020Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences
  7. 2022Causal Imitation Learning with Unobserved Confounders
  8. 2019Reinforced Imitation in Heterogeneous Action Space