# Deconfounding Imitation Learning with Variational Inference

arXiv: https://arxiv.org/abs/2211.02667 · HTML: https://mlsanity.com/paper/2211.02667/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 29 of its 54 references that are in the map.
- First posted on arXiv 4 Nov 2022.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 29 of 29)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [RMA: Rapid Motor Adaptation for Legged Robots](https://mlsanity.com/paper/2107.04034/) (2021)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables](https://mlsanity.com/paper/1903.08254/) (2019)
- [Learning to reinforcement learn](https://mlsanity.com/paper/1611.05763/) (2016)
- [Exploring the Limitations of Behavior Cloning for Autonomous Driving](https://mlsanity.com/paper/1904.08980/) (2019)
- [Causal Confusion in Imitation Learning](https://mlsanity.com/paper/1905.11979/) (2019)
- [Learning Agile Robotic Locomotion Skills by Imitating Animals](https://mlsanity.com/paper/2004.00784/) (2020)
- [One-Shot Imitation Learning](https://mlsanity.com/paper/1703.07326/) (2017)
- [VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning](https://mlsanity.com/paper/1910.08348/) (2019)
- [DART: Noise Injection for Robust Imitation Learning](https://mlsanity.com/paper/1703.09327/) (2017)
- [Deep Kalman Filters](https://mlsanity.com/paper/1511.05121/) (2015)
- [Fighting Copycat Agents in Behavioral Cloning from Observation Histories](https://mlsanity.com/paper/2010.14876/) (2020)
- [On the Expressivity of Markov Reward](https://mlsanity.com/paper/2111.00876/) (2021)
- [Feedback in Imitation Learning: The Three Regimes of Covariate Shift](https://mlsanity.com/paper/2102.02872/) (2021)
- [Shaking the foundations: delusions in sequence models for interaction and control](https://mlsanity.com/paper/2110.10819/) (2021)
- [Environment Probing Interaction Policies](https://mlsanity.com/paper/1907.11740/) (2019)
- [Sequence Model Imitation Learning with Unobserved Contexts](https://mlsanity.com/paper/2208.02225/) (2022)
- [Causal Imitation Learning under Temporally Correlated Noise](https://mlsanity.com/paper/2202.01312/) (2022)
- [Sequential Causal Imitation Learning with Unobserved Confounders](https://mlsanity.com/paper/2208.06276/) (2022)
- [Causally Correct Partial Models for Reinforcement Learning](https://mlsanity.com/paper/2002.02836/) (2020)
- [Fighting Fire with Fire: Avoiding DNN Shortcuts through Priming](https://mlsanity.com/paper/2206.10816/) (2022)

## Built on by (most cited first, 5 of 5)

- [Causal Semantic Communication for Digital Twins: A Generalizable Imitation Learning Approach](https://mlsanity.com/paper/2304.12502/) (2023)
- [FoMo Rewards: Can we cast foundation models as reward functions?](https://mlsanity.com/paper/2312.03881/) (2023)
- [IGDrivSim: A Benchmark for the Imitation Gap in Autonomous Driving](https://mlsanity.com/paper/2411.04653/) (2024)
- [A Bayesian Solution To The Imitation Gap](https://mlsanity.com/paper/2407.00495/) (2024)
- [Scalable Causal Imitation Learning](https://mlsanity.com/paper/2607.17003/) (2026)

## Abstract (arXiv)

Standard imitation learning can fail when the expert demonstrators have different sensory inputs than the imitating agent. This is because partial observability gives rise to hidden confounders in the causal graph. In previous work, to work around the confounding problem, policies have been trained using query access to the expert's policy or inverse reinforcement learning (IRL). However, both approaches have drawbacks as the expert's policy may not be available and IRL can be unstable in practice. Instead, we propose to train a variational inference model to infer the expert's latent information and use it to train a latent-conditional policy. We prove that using this method, under strong assumptions, the identification of the correct imitation learning policy is theoretically possible from expert demonstrations alone. In practice, we focus on a setting with less strong assumptions where we use exploration data for learning the inference model. We show in theory and practice that this algorithm converges to the correct interventional policy, solves the confounding issue, and can under certain assumptions achieve an asymptotically optimal imitation performance.