mlsanityData to 14 Oct 2026

Action-Sufficient State Representation Learning for Control with Structural Constraints

arXiv 2110.05721 ↗First posted 12 Oct 2021cs.LG cs.AIcited by 31

Not on the map. Why?

Abstract (arXiv)

Perceived signals in real-world scenarios are usually high-dimensional and noisy, and finding and using their representation that contains essential and sufficient information required by downstream decision-making tasks will help improve computational efficiency and generalization ability in the tasks. In this paper, we focus on partially observable environments and propose to learn a minimal set of state representations that capture sufficient information for decision-making, termed textitAction-Sufficient state Representations (ASRs). We build a generative environment model for the structural relationships among variables in the system and present a principled way to characterize ASRs based on structural constraints and the goal of maximizing cumulative reward in policy learning. We then develop a structured sequential Variational Auto-Encoder to estimate the environment model and extract ASRs. Our empirical results on CarRacing and VizDoom demonstrate a clear advantage of learning and using ASRs for policy learning. Moreover, the estimated environment model and ASRs allow learning behaviors from imagined outcomes in the compact latent space to improve sample efficiency.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Builds on 32 of its 53 references that are in the map. First posted on arXiv 12 Oct 2021.

Builds on

32 of 53 references.

References · 32

  1. 2015Continuous control with deep reinforcement learningcited by 3,407
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  3. 2017The Kinetics Human Action Video Datasetcited by 1,909
  4. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  5. 2018World Modelscited by 1,513
  6. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  7. 2020Mastering Atari with Discrete World Modelscited by 1,007
  8. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  9. 2019Model-Based Reinforcement Learning for Ataricited by 502
  10. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 465
  11. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
  12. 2016The CMA Evolution Strategy: A Tutorialcited by 377
20 more
  1. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 274
  2. 2019Causality for Machine Learningcited by 243
  3. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 231
  4. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 207
  5. 2016ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learningcited by 179
  6. 2015Deep Kalman Filterscited by 166
  7. 2019Causal Discovery from Heterogeneous/Nonstationary Data with Independent Changescited by 166
  8. 2016Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Datacited by 156
  9. 2017Self-Supervised Visual Planning with Temporal Skip Connectionscited by 139
  10. 2019Scalable methods for computing state similarity in deterministic Markov Decision Processescited by 133
  11. 2018State Representation Learning for Control: An Overviewcited by 127
  12. 2017The Consciousness Priorcited by 107
  13. 2016Deep Successor Reinforcement Learningcited by 72
  14. 2018Learning Actionable Representations with Goal-Conditioned Policiescited by 54
  15. 2018Temporal Difference Variational Auto-Encodercited by 49
  16. 2019Causal Discovery and Forecasting in Nonstationary Environments with State-Space Modelscited by 45
  17. 2020Domain Adaptation as a Problem of Inference on Graphical Modelscited by 36
  18. 2019Learning Causal State Representations of Partially Observable Environmentscited by 30
  19. 2020Predictive Coding for Locally-Linear Controlcited by 17
  20. 2017Independently Controllable Factorscited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 5; 2024: 13; 2025: 7; 2026: 5.

31 papers.

Sort:

By area:

  1. 2023A Survey on Causal Reinforcement Learningcited by 45
  2. 2022MoCoDA: Model-based Counterfactual Data Augmentationcited by 44
  3. 2023Learning World Models with Identifiable Factorizationcited by 23
  4. 2023Causal Reinforcement Learning: A Surveycited by 23
  5. 2023Interpretable Reward Redistribution in Reinforcement Learning: A Causal Approachcited by 19
  6. 2024ACE : Off-Policy Actor-Critic with Causality-Aware Entropy Regularizationcited by 16
  7. 2024Building Minimal and Reusable Causal State Abstractions for Reinforcement Learningcited by 11
  8. 2024Fine-Grained Causal Dynamics Learning with Quantization for Improving Robustness in Reinforcement Learningcited by 11
  9. 2024AD3: Implicit Action is the Key for World Models to Distinguish the Diverse Visual Distractorscited by 10
  10. 2023MACCA: Offline Multi-agent Reinforcement Learning with Causal Credit Assignmentcited by 6
  11. 2025Towards Empowerment Gain through Causal Structure Learning in Model-Based RLcited by 5
  12. 2025Modeling Unseen Environments with Language-guided Composable Causal Components in Reinforcement Learningcited by 4
19 more
  1. 2024Knowing What Not to Do: Leverage Language Model Insights for Action Space Pruning in Multi-agent Reinforcement Learningcited by 3
  2. 2024BECAUSE: Bilinear Causal Representation for Generalizable Offline Model-based Reinforcement Learningcited by 3
  3. 2024Towards Generalizable Reinforcement Learning via Causality-Guided Self-Adaptive Representationscited by 2
  4. 2025Policy-Guided Causal State Representation for Offline Reinforcement Learning Recommendationcited by 2
  5. 2026Low-Rank Dynamics-Effective Latent Carriers for Counterfactual Rollout in Learned World Modelscited by 2
  6. 2024Why Online Reinforcement Learning is Causalcited by 1
  7. 2024On Causally Disentangled State Representation Learning for Reinforcement Learning based Recommender Systemscited by 1
  8. 2024Boosting Efficiency in Task-Agnostic Exploration through Causal Knowledgecited by 1
  9. 2024Identifiable Representation and Model Learning for Latent Dynamic Systemscited by 1
  10. 2025How Well Do Latent World Models Understand Partially Observable Safety Constraints?cited by 1
  11. 2026Factored Causal Representation Learning for Robust Reward Modeling in RLHFcited by 1
  12. 2026Action-Sufficient Goal Representationscited by 1
  13. 2024Disentangled Representations for Causal Cognitioncited by 0
  14. 2024Rethinking State Disentanglement in Causal Reinforcement Learningcited by 0
  15. 2025Q-function Decomposition with Intervention Semantics with Factored Action Spacescited by 0
  16. 2025Observations Meet Actions: Learning Control-Sufficient Representations for Robust Policy Generalizationcited by 0
  17. 2025Learning Causal States Under Partial Observability and Perturbationcited by 0
  18. 2026What a World Model Represents Is Three Questionscited by 0
  19. 2026AD-WM: Action-Discriminative World Models for Counterfactual Model Predictive Controlcited by 0