mlsanityData to 7 Oct 2026

Learning Causally Invariant Reward Functions from Diverse Demonstrations

arXiv 2409.08012 ↗First posted 12 Sep 2024cs.LG cs.AIcited by 2

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

Inverse reinforcement learning methods aim to retrieve the reward function of a Markov decision process based on a dataset of expert demonstrations. The commonplace scarcity and heterogeneous sources of such demonstrations can lead to the absorption of spurious correlations in the data by the learned reward function. Consequently, this adaptation often exhibits behavioural overfitting to the expert data set when a policy is trained on the obtained reward function under distribution shift of the environment dynamics. In this work, we explore a novel regularization approach for inverse reinforcement learning methods based on the causal invariance principle with the goal of improved reward function generalization. By applying this regularization to both exact and approximate formulations of the learning task, we demonstrate superior policy performance when trained using the recovered reward functions in a transfer setting

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 29 of its 52 references that are in the map. First posted on arXiv 12 Sep 2024.

Builds on

29 of 52 references.

Closest ancestors · 3

Same direction, newest first

  1. 2020Inverse Reinforcement Learning via Matching of Optimality Profiles18 Nov
  2. 2020f-IRL: Inverse Reinforcement Learning via State Marginal Matching9 Nov
  3. 2020Robust Inverse Reinforcement Learning under Transition Dynamics Mismatch2 Jul

Other references · 26

  1. 2022Generative Adversarial Networkscited by 5,603
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2017Improved Training of Wasserstein GANscited by 2,223
  4. 2019Invariant Risk Minimizationcited by 1,832
  5. 2016Generative Adversarial Imitation Learningcited by 1,576
  6. 2015Causal inference using invariant prediction: identification and confidence intervalscited by 533
  7. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  8. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  9. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  10. 2019Causal Confusion in Imitation Learningcited by 286
  11. 2021Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalizationcited by 220
  12. 2020Invariant Risk Minimization Gamescited by 188
14 more
  1. 2017Spectral Norm Regularization for Improving the Generalizability of Deep Learningcited by 178
  2. 2015Maximum Entropy Deep Inverse Reinforcement Learningcited by 173
  3. 2016A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Modelscited by 167
  4. 2017Invariant Causal Prediction for Nonlinear Modelscited by 140
  5. 2018Anchor regression: heterogeneous data meets causalitycited by 116
  6. 2020Invariant Causal Prediction for Block MDPscited by 104
  7. 2019Observational Overfitting in Reinforcement Learningcited by 90
  8. 2021Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gapcited by 80
  9. 2021Off-Policy Imitation Learning from Observationscited by 70
  10. 2019Task-Relevant Adversarial Imitation Learningcited by 45
  11. 2020Invariant Policy Optimization: Towards Stronger Generalization in Reinforcement Learningcited by 37
  12. 2022Causal Imitation Learning under Temporally Correlated Noisecited by 28
  13. 2018Kernel Exponential Family Estimation via Doubly Dual Embeddingcited by 16
  14. 2023Invariant Causal Imitation Learning for Generalizable Policiescited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2026Factored Causal Representation Learning for Robust Reward Modeling in RLHFcited by 1
  2. 2025Hallucination-Resistant Relation Extraction via Dependency-Aware Sentence Simplification and Two-tiered Hierarchical Refinementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2019Meta-Inverse Reinforcement Learning with Probabilistic Context Variables
  2. 2025Statistical analysis of Inverse Entropy-regularized Reinforcement Learning
  3. 2020Efficient Exploration of Reward Functions in Inverse Reinforcement Learning via Bayesian Optimization
  4. 2021Versatile Inverse Reinforcement Learning via Cumulative Rewards
  5. 2018Learning a Prior over Intent via Meta-Inverse Reinforcement Learning
  6. 2023LS-IQ: Implicit Reward Regularization for Inverse Reinforcement Learning
  7. 2021Identifiability in inverse reinforcement learning
  8. 2026Reward Transfer from Inverse Reinforcement Learning: A Coupled Minimax Approach