mlsanityData to 7 Oct 2026

Offline Reinforcement Learning with Causal Structured World Models

arXiv 2206.01474 ↗First posted 3 Jun 2022cs.LG stat.MLcited by 19

On the map: World models → Latent dynamics models and model-based RL

also in 1 more
Abstract (arXiv)

Model-based methods have recently shown promising for offline reinforcement learning (RL), aiming to learn good policies from historical data without interacting with the environment. Previous model-based offline RL methods learn fully connected nets as world-models that map the states and actions to the next-step states. However, it is sensible that a world-model should adhere to the underlying causal effect such that it will support learning an effective policy generalizing well in unseen states. In this paper, We first provide theoretical results that causal world-models can outperform plain world-models for offline RL by incorporating the causal structure into the generalization error bound. We then propose a practical algorithm, oFfline mOdel-based reinforcement learning with CaUsal Structure (FOCUS), to illustrate the feasibility of learning and leveraging causal structure in offline RL. Experimental results on two benchmarks show that FOCUS reconstructs the underlying causal structure accurately and robustly. Consequently, it performs better than the plain model-based offline RL algorithms and other causal model-based RL algorithms.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 11 of its 28 references that are in the map. First posted on arXiv 3 Jun 2022.

Builds on

11 of 28 references.

Closest ancestors · 2

Same direction, newest first

  1. 2018Recurrent World Models Facilitate Policy Evolution4 Sep
  2. 2018Model-Ensemble Trust-Region Policy Optimization28 Feb

Other references · 9

  1. 2019Solving Rubik's Cube with a Robot Handcited by 699
  2. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  3. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  4. 2019Causal Confusion in Imitation Learningcited by 286
  5. 2019A Meta-Transfer Objective for Learning to Disentangle Causal Mechanismscited by 158
  6. 2019Learning Neural Causal Models from Unknown Interventionscited by 113
  7. 2017Building machines that adapt and compute like brainscited by 99
  8. 2020Error Bounds of Imitating Policies and Environmentscited by 90
  9. 2018Virtual-Taobao: Virtualizing Real-world Online Retail Environment for Reinforcement Learningcited by 80

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 6; 2024: 5; 2025: 2; 2026: 3.

19 papers.

Sort:

By area:

  1. 2022A Survey on Model-based Reinforcement Learningcited by 65
  2. 2023A Survey on Causal Reinforcement Learningcited by 43
  3. 2022Generalizing Goal-Conditioned Reinforcement Learning with Variational Causal Reasoningcited by 33
  4. 2024The Essential Role of Causality in Foundation World Models for Embodied AIcited by 32
  5. 2023Causal Reinforcement Learning: A Surveycited by 22
  6. 2023On the Opportunities and Challenges of Offline Reinforcement Learning for Recommender Systemscited by 10
  7. 2024Any-step Dynamics Model Improves Future Predictions for Online and Offline Reinforcement Learningcited by 10
  8. 2023Offline Imitation Learning with Variational Counterfactual Reasoningcited by 7
  9. 2022Model-based Reinforcement Learning with Multi-step Plan Value Estimationcited by 6
  10. 2023Sim2Rec: A Simulator-based Decision-making Approach to Optimize Real-World Long-term User Engagement in Sequential Recommender Systemscited by 5
  11. 2024Learning Causal Dynamics Models in Object-Oriented Environmentscited by 3
  12. 2024BECAUSE: Bilinear Causal Representation for Generalizable Offline Model-based Reinforcement Learningcited by 3
7 more
  1. 2023Beware of Instantaneous Dependence in Reinforcement Learningcited by 1
  2. 2024Boosting Efficiency in Task-Agnostic Exploration through Causal Knowledgecited by 1
  3. 2025Efficient and Generalizable Environmental Understanding for Visual Navigationcited by 1
  4. 2025Hierarchical Reinforcement Learning with Targeted Causal Interventionscited by 0
  5. 2026Transferable Delay-Aware Reinforcement Learning via Implicit Causal Graph Modelingcited by 0
  6. 2026Learning Implicit Causal World Models from Multi-Agent Demonstrationscited by 0
  7. 2026I Act Therefore I Am: When Is JEPA's Action-Conditioning Enough to Learn Causal Mechanisms?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Causal prompting model-based offline reinforcement learning
  2. 2023A Survey on Offline Model-Based Reinforcement Learning
  3. 2024Why Online Reinforcement Learning is Causal
  4. 2025Policy-Driven World Model Adaptation for Robust Offline Model-based Reinforcement Learning
  5. 2022Domain Generalization for Robust Model-Based Offline Reinforcement Learning
  6. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
  7. 2024Out-of-Distribution Adaptation in Offline RL: Counterfactual Reasoning via Causal Normalizing Flows
  8. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory