mlsanityData to 14 Oct 2026

From Causal Graphs to Markovian States: Multi-Order State Exposure for Deep RL

arXiv 2605.07057 ↗First posted 8 May 2026cs.LGcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Online reinforcement learning (RL) relies on the Markov property for guaranteed performance, but real-world applications often lack well-defined states given raw observed variables. While causal RL has attracted growing interest, existing work typically assumes Markovian states are provided and uses causality only to accelerate learning. This leaves a fundamental gap: given a longitudinal causal graph over observed variables, how does one construct MDP states that provably satisfy the Markov property? We provide a procedure that constructs a provably minimal such state. In deep RL, the minimal representation alone empirically fails to improve performance, indicating that neural networks cannot automatically exploit Markovian minimality. We therefore propose MOSE (Multi-Order State Exposure), which exposes a single shared Q-function to history-based state constructions of orders 1 through W simultaneously. An attention-based selection layer scores the dimensions of this combined input by their estimated relevance to value-function estimation and selects the high-scoring ones. Causal MOSE further constrains the selection layer to retain every variable of the minimal state, guaranteeing sufficiency while leaving the model free to admit useful redundancy. MOSE consistently outperforms both the minimal state construction and single-window policies on common benchmarks and synthetic datasets. Including the minimal representation alongside MOSE can further improve performance. Our results establish a core principle for causal deep RL: minimal sufficiency is not enough. Controlled redundancy gives the model flexibility to learn its state representation end-to-end. Causal state information provides structural guidance that can make this learning more efficient.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 56 of its 92 references that are in the map. First posted on arXiv 8 May 2026.

Builds on

56 of 92 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026Confounding Robust Continuous Control via Automatic Reward Shaping10 Feb
  2. 2025Confounding Robust Deep Reinforcement Learning: A Causal Approach24 Oct
  3. 2025Automatic Reward Shaping from Confounded Offline Data16 May
  4. 2024Building Minimal and Reusable Causal State Abstractions for Reinforcement Learning23 Jan
  5. 2024Bridging State and History Representations: Understanding Self-Predictive RL17 Jan

Other references · 51

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  4. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  5. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  6. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  7. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  8. 2019Model-Based Reinforcement Learning for Ataricited by 502
  9. 2020Reinforcement Learning with Augmented Datacited by 468
  10. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 465
  11. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 459
  12. 2018Quantifying Generalization in Reinforcement Learningcited by 352
38 more
  1. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 303
  2. 2020Decoupling Representation Learning from Reinforcement Learningcited by 269
  3. 2017Noisy Networks for Explorationcited by 266
  4. 2013Causal Discovery with Continuous Additive Noise Modelscited by 255
  5. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
  6. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  7. 2017Detecting causal associations in large nonlinear time series datasetscited by 244
  8. 2020DYNOTEARS: Structure Learning from Time-Series Datacited by 200
  9. 2022Online Decision Transformercited by 196
  10. 2018A Study on Overfitting in Deep Reinforcement Learningcited by 174
  11. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 174
  12. 2022Interventional Causal Representation Learningcited by 142
  13. 2020Discovering contemporaneous and lagged causal relations in autocorrelated nonlinear time series datasetscited by 137
  14. 2018State Representation Learning for Control: An Overviewcited by 127
  15. 2023Nonparametric Identifiability of Causal Representations from Unknown Interventionscited by 106
  16. 2020Invariant Causal Prediction for Block MDPscited by 105
  17. 2016Causal Bandits: Learning Good Interventions via Causal Inferencecited by 101
  18. 2020Amortized Causal Discovery: Learning to Infer Causal Graphs from Time-Series Datacited by 100
  19. 2024Causal Representation Learning from Multiple Distributions: A General Settingcited by 68
  20. 2022Causal Dynamics Learning for Task-Independent State Abstractioncited by 65
  21. 2022Look Closer: Bridging Egocentric and Third-Person Views with Transformers for Robotic Manipulationcited by 49
  22. 2021Interpretable Models for Granger Causality Using Self-explaining Neural Networkscited by 45
  23. 2021Neural graphical modelling in continuous-time: consistency guarantees and algorithmscited by 42
  24. 2017Identifying Best Interventions through Online Importance Samplingcited by 39
  25. 2020Budgeted and Non-budgeted Causal Banditscited by 38
  26. 2020Can Increasing Input Dimensionality Improve Deep Reinforcement Learning?cited by 37
  27. 2021Causal Bandits with Unknown Graph Structurecited by 37
  28. 2023A Survey on Causal Discovery Methods for I.I.D. and Time Series Datacited by 35
  29. 2023General Identifiability and Achievability for Causal Representation Learningcited by 34
  30. 2021NTS-NOTEARS: Learning Nonparametric DBNs With Prior Knowledgecited by 33
  31. 2020Does the Markov Decision Process Fit the Data: Testing for the Markov Property in Sequential Decision Makingcited by 30
  32. 2021Unsupervised Learning of Visual 3D Keypoints for Controlcited by 30
  33. 2025Rethinking Progression of Memory State in Robotic Manipulation: An Object-Centric Perspectivecited by 20
  34. 2025Robust Reward Modeling via Causal Rubricscited by 19
  35. 2023Causal discovery for time series from multiple datasets with latent contextscited by 16
  36. 2019Multi-View Reinforcement Learningcited by 12
  37. 2023Testing for the Markov Property in Time Series via Deep Conditional Generative Learningcited by 9
  38. 2024Harnessing Causality in Reinforcement Learning With Bagged Decision Timescited by 5

50 most cited of 51 All 51 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Rethinking State Disentanglement in Causal Reinforcement Learning
  2. 2021Provably Efficient Representation Selection in Low-rank Markov Decision Processes: From Online to Offline RL
  3. 2022Offline Reinforcement Learning with Causal Structured World Models
  4. 2026Breaking the Capability Ceiling of LLM Post-Training by Reintroducing Markov States
  5. 2025Learning Causal States Under Partial Observability and Perturbation
  6. 2025Dynamic Sparsity: Challenging Common Sparsity Assumptions for Learning World Models in Robotic Reinforcement Learning Benchmarks
  7. 2025Quantifying First-Order Markov Violations in Noisy Reinforcement Learning: A Causal Discovery Approach
  8. 2026Belief-State RWKV for Reinforcement Learning under Partial Observability