mlsanityData to 7 Oct 2026

On Covariate Shift of Latent Confounders in Imitation and Reinforcement Learning

arXiv 2110.06539 ↗First posted 13 Oct 2021cs.LG cs.AI cs.ROcited by 14

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

We consider the problem of using expert data with unobserved confounders for imitation and reinforcement learning. We begin by defining the problem of learning from confounded expert data in a contextual MDP setup. We analyze the limitations of learning from such data with and without external reward, and propose an adjustment of standard imitation learning algorithms to fit this setup. We then discuss the problem of distribution shift between the expert data and the online environment when the data is only partially observable. We prove possibility and impossibility results for imitation learning under arbitrary distribution shift of the missing covariates. When additional external reward is provided, we propose a sampling procedure that addresses the unknown shift and prove convergence to an optimal solution. Finally, we validate our claims empirically on challenging assistive healthcare and recommender system simulation tasks.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 24 of its 43 references that are in the map. First posted on arXiv 13 Oct 2021.

Builds on

24 of 43 references.

Closest ancestors · 3

Same direction, newest first

  1. 2019Imitation Learning via Off-Policy Distribution Matching10 Dec
  2. 2019Imitation Learning as f-Divergence Minimization30 May
  3. 2019Causal Confusion in Imitation Learning28 May

Other references · 21

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  4. 2016Generative Adversarial Imitation Learningcited by 1,576
  5. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  6. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  7. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  8. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
  9. 2015Contextual Markov Decision Processescited by 213
  10. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 213
  11. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  12. 2019Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Modelscited by 114
9 more
  1. 2016Causal Bandits: Learning Good Interventions via Causal Inferencecited by 100
  2. 2019RecSim: A Configurable Simulation Platform for Recommender Systemscited by 91
  3. 2021Reward is enough for convex MDPscited by 82
  4. 2019Off-Policy Evaluation in Partially Observable Environmentscited by 65
  5. 2020Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confoundingcited by 60
  6. 2020Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learningcited by 56
  7. 2019Assistive Gym: A Physics Simulation Framework for Assistive Roboticscited by 44
  8. 2020Provably Efficient Causal Reinforcement Learning with Confounded Observational Datacited by 40
  9. 2020Bandits with Partially Observable Confounded Datacited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 4; 2024: 3; 2025: 3; 2026: 1.

14 papers.

Sort:

By area:

  1. 2023A Survey on Causal Reinforcement Learningcited by 43
  2. 2022Sequence Model Imitation Learning with Unobserved Contextscited by 34
  3. 2023Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlationcited by 25
  4. 2024Mitigating Covariate Shift in Imitation Learning for Autonomous Vehicles Using Latent Space Generative World Modelscited by 22
  5. 2023Chain-of-Thought Predictive Controlcited by 18
  6. 2022Discriminator-Guided Model-Based Offline Imitation Learningcited by 17
  7. 2022Planning and Learning with Adaptive Lookaheadcited by 8
  8. 2023Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confoundingcited by 7
  9. 2024DynaMITE-RL: A Dynamic Model for Improved Temporal Meta-Reinforcement Learningcited by 3
  10. 2025TAR: Teacher-Aligned Representations via Contrastive Learning for Quadrupedal Locomotioncited by 3
  11. 2024Benchmarks for Reinforcement Learning with Biased Offline Data and Imperfect Simulatorscited by 0
  12. 2025Breaking Habits: On the Role of the Advantage Function in Learning Causal State Representationscited by 0
2 more
  1. 2025Confounded Causal Imitation Learning with Instrumental Variablescited by 0
  2. 2026Scalable Causal Imitation Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Deconfounding Imitation Learning with Variational Inference
  2. 2021Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage
  3. 2022Causal Imitation Learning with Unobserved Confounders
  4. 2021Feedback in Imitation Learning: The Three Regimes of Covariate Shift
  5. 2025Causal Imitation Learning under Expert-Observable and Expert-Unobservable Confounding
  6. 2022Minimax Optimal Online Imitation Learning via Replay Estimation
  7. 2025Using Non-Expert Data to Robustify Imitation Learning via Offline Reinforcement Learning
  8. 2021Imitation Learning by Reinforcement Learning