mlsanityData to 7 Oct 2026

BECAUSE: Bilinear Causal Representation for Generalizable Offline Model-based Reinforcement Learning

arXiv 2407.10967 ↗First posted 15 Jul 2024cs.LG cs.AIcited by 3

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline model-based reinforcement learning (MBRL) enhances data efficiency by utilizing pre-collected datasets to learn models and policies, especially in scenarios where exploration is costly or infeasible. Nevertheless, its performance often suffers from the objective mismatch between model and policy learning, resulting in inferior performance despite accurate model predictions. This paper first identifies the primary source of this mismatch comes from the underlying confounders present in offline data for MBRL. Subsequently, we introduce BilinEar CAUSal rEpresentation~(BECAUSE), an algorithm to capture causal representation for both states and actions to reduce the influence of the distribution shift, thus mitigating the objective mismatch problem. Comprehensive evaluations on 18 tasks that vary in data quality and environment context demonstrate the superior performance of BECAUSE over existing offline RL algorithms. We show the generalizability and robustness of BECAUSE under fewer samples or larger numbers of confounders. Additionally, we offer theoretical analysis of BECAUSE to prove its error bound and sample efficiency when integrating causal representation into offline MBRL.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 67 of its 92 references that are in the map. First posted on arXiv 15 Jul 2024.

Builds on

67 of 92 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024Reward-Relevance-Filtered Linear Offline Reinforcement Learning23 Jan
  2. 2023Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confounding1 Jun
  3. 2022MoCoDA: Model-based Counterfactual Data Augmentation20 Oct
  4. 2022A Unified Framework for Alternating Offline Model Training and Policy Learning12 Oct
  5. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexity11 Aug
  6. 2022Offline Reinforcement Learning with Causal Structured World Models3 Jun
  7. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage13 Jul
  8. 2021Provably Efficient Representation Selection in Low-rank Markov Decision Processes: From Online to Offline RL22 Jun
  9. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  10. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  11. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  12. 2020MOPO: Model-based Offline Policy Optimization27 May
2 more
  1. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 53

  1. 2017Modeling Relational Data with Graph Convolutional Networkscited by 1,684
  2. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  3. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  4. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 428
  5. 2019Causal Confusion in Imitation Learningcited by 286
  6. 2019Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biasescited by 279
  7. 2020Differentiable Causal Discovery from Interventional Datacited by 218
  8. 2019Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Boundcited by 199
  9. 2017Spectral Norm Regularization for Improving the Generalizability of Deep Learningcited by 178
  10. 2019Explainable Reinforcement Learning Through a Causal Lenscited by 177
  11. 2019Causal Discovery with Reinforcement Learningcited by 145
  12. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 138
38 more
  1. 2019Learning Neural Causal Models from Unknown Interventionscited by 113
  2. 2020Invariant Causal Prediction for Block MDPscited by 104
  3. 2020Learning Latent Space Energy-Based Prior Modelcited by 104
  4. 2020Objective Mismatch in Model-based Reinforcement Learningcited by 87
  5. 2018Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Searchcited by 81
  6. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
  7. 2020Causal Discovery in Physical Systems from Videoscited by 77
  8. 2018Characterizing and Learning Equivalence Classes of Causal DAGs under Interventionscited by 76
  9. 2021Causal Influence Detection for Improving Efficiency in Reinforcement Learningcited by 70
  10. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
  11. 2022Denoised MDPs: Learning World Models Better Than the World Itselfcited by 66
  12. 2022Causal Dynamics Learning for Task-Independent State Abstractioncited by 64
  13. 2021A Survey of Deep RL and IL for Autonomous Driving Policy Learningcited by 59
  14. 2021Learning Task Informed Abstractionscited by 58
  15. 2021AdaRL: What, Where, and How to Adapt in Transfer Reinforcement Learningcited by 46
  16. 2021Energy-Based Open-World Uncertainty Modeling for Confidence Calibrationcited by 46
  17. 2020Goal-Aware Prediction: Learning to Model What Matterscited by 45
  18. 2020Provably Efficient Causal Reinforcement Learning with Confounded Observational Datacited by 40
  19. 2021Learning Neural Causal Models with Active Interventionscited by 40
  20. 2022Causal Imitation Learning with Unobserved Confounderscited by 40
  21. 2021Mismatched No More: Joint Model-Policy Optimization for Model-Based RLcited by 35
  22. 2022Generalizing Goal-Conditioned Reinforcement Learning with Variational Causal Reasoningcited by 33
  23. 2022Learning Iterative Reasoning through Energy Minimizationcited by 32
  24. 2019Causal Induction from Visual Observations for Goal Directed Taskscited by 31
  25. 2021Action-Sufficient State Representation Learning for Control with Structural Constraintscited by 31
  26. 2022Factored Adaptation for Non-Stationary Reinforcement Learningcited by 30
  27. 2023Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlationcited by 25
  28. 2023Learning World Models with Identifiable Factorizationcited by 23
  29. 2023What Went Wrong? Closing the Sim-to-Real Gap via Differentiable Causal Discoverycited by 22
  30. 2021Causal Reasoning in Simulation for Structure and Transfer Learning of Robot Manipulation Policiescited by 20
  31. 2019Gradient-Aware Model-based Policy Searchcited by 19
  32. 2023Interpretable Reward Redistribution in Reinforcement Learning: A Causal Approachcited by 17
  33. 2018Fast Conditional Independence Test for Vector Variables with Large Sample Sizescited by 16
  34. 2024ACE : Off-Policy Actor-Critic with Causality-Aware Entropy Regularizationcited by 16
  35. 2023Learning Dynamic Attribute-factored World Models for Efficient Multi-object Reinforcement Learningcited by 15
  36. 2023Invariant Causal Imitation Learning for Generalizable Policiescited by 15
  37. 2020Resolving Spurious Correlations in Causal Models of Environments via Interventionscited by 14
  38. 2023Computably Continuous Reinforcement-Learning Objectives are PAC-learnablecited by 13

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Learning Interactive World Model for Object-Centric Reinforcement Learningcited by 11
  2. 2025Curious Causality-Seeking Agents Learn Meta Causal Worldcited by 4
  3. 2026Factorized Spectral Representations for Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Representation Matters: Offline Pretraining for Sequential Decision Making
  2. 2025CausalCOMRL: Context-Based Offline Meta-Reinforcement Learning with Causal Representation
  3. 2024Out-of-Distribution Adaptation in Offline RL: Counterfactual Reasoning via Causal Normalizing Flows
  4. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
  5. 2024Causal prompting model-based offline reinforcement learning
  6. 2025Policy-Driven World Model Adaptation for Robust Offline Model-based Reinforcement Learning
  7. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  8. 2024Offline Model-Based Reinforcement Learning with Anti-Exploration