mlsanityData to 14 Oct 2026

Building Minimal and Reusable Causal State Abstractions for Reinforcement Learning

arXiv 2401.12497 ↗First posted 23 Jan 2024cs.AI cs.LG cs.ROcited by 11

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Two desiderata of reinforcement learning (RL) algorithms are the ability to learn from relatively little experience and the ability to learn policies that generalize to a range of problem specifications. In factored state spaces, one approach towards achieving both goals is to learn state abstractions, which only keep the necessary variables for learning the tasks at hand. This paper introduces Causal Bisimulation Modeling (CBM), a method that learns the causal relationships in the dynamics and reward functions for each task to derive a minimal, task-specific abstraction. CBM leverages and improves implicit modeling to train a high-fidelity causal dynamics model that can be reused for all tasks in the same environment. Empirical validation on manipulation environments and Deepmind Control Suite reveals that CBM's learned implicit dynamics models identify the underlying causal relationships and state abstractions more accurately than explicit ones. Furthermore, the derived state abstractions allow a task learner to achieve near-oracle levels of sample efficiency and outperform baselines on all tasks.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 23 of its 36 references that are in the map. First posted on arXiv 23 Jan 2024.

Builds on

23 of 36 references.

References · 23

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  4. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  5. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  6. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  7. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 511
  8. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
  9. 2019Sliced Score Matching: A Scalable Approach to Density and Score Estimationcited by 384
  10. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 297
  11. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 219
  12. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 156
11 more
  1. 2021Tianshou: a Highly Modularized Deep Reinforcement Learning Librarycited by 143
  2. 2020Invariant Causal Prediction for Block MDPscited by 105
  3. 2022Causal Dynamics Learning for Task-Independent State Abstractioncited by 65
  4. 2021Learning Task Informed Abstractionscited by 58
  5. 2020ContactNets: Learning Discontinuous Contact Dynamics with Smooth, Implicit Representationscited by 38
  6. 2018Deep Energy Estimator Networkscited by 34
  7. 2022Generalizing Goal-Conditioned Reinforcement Learning with Variational Causal Reasoningcited by 34
  8. 2021Action-Sufficient State Representation Learning for Control with Structural Constraintscited by 31
  9. 2019Learning Causal State Representations of Partially Observable Environmentscited by 30
  10. 2021Decomposed Mutual Information Estimation for Contrastive Representation Learningcited by 25
  11. 2021Joint Energy-based Model Training for Better Calibrated Natural Language Understanding Modelscited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 4; 2026: 4.

11 papers.

  1. 2025Towards Empowerment Gain through Causal Structure Learning in Model-Based RLcited by 5
  2. 2025Curious Causality-Seeking Agents Learn Meta Causal Worldcited by 4
  3. 2024Why Online Reinforcement Learning is Causalcited by 1
  4. 2024On the Sample Efficiency of Abstractions and Potential-Based Reward Shaping in Reinforcement Learningcited by 1
  5. 2025Dynamic Sparsity: Challenging Common Sparsity Assumptions for Learning World Models in Robotic Reinforcement Learning Benchmarkscited by 1
  6. 2024Learning telic-controllable state representationscited by 0
  7. 2025Context-Sensitive Abstractions for Reinforcement Learning with Parameterized Actionscited by 0
  8. 2026From Causal Graphs to Markovian States: Multi-Order State Exposure for Deep RLcited by 0
  9. 2026SAT-RTS: A systematic framework for tactical knowledge extraction and visualization-based analysis in real-time strategy gamescited by 0
  10. 2026Property-driven Causal Abstractions for Markov Decision Processescited by 0
  11. 2026A Unifying Perspective on Causal World Models: From Observations to Representations to Structurecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Causal-Paced Deep Reinforcement Learning
  2. 2026CausalGDP: Causality-Guided Diffusion Policies for Reinforcement Learning
  3. 2025Unifying Causal Reinforcement Learning: Survey, Taxonomy, Algorithms and Applications
  4. 2026An Introduction to Causal Reinforcement Learning
  5. 2024Learning Causal Dynamics Models in Object-Oriented Environments
  6. 2025Learning Nonlinear Causal Reductions to Explain Reinforcement Learning Policies
  7. 2024Fine-Grained Causal Dynamics Learning with Quantization for Improving Robustness in Reinforcement Learning
  8. 2026Compositional Behavioral Semantics for State Abstraction in Reinforcement Learning