From Pixels to Factors: Learning Independently Controllable State Variables for Reinforcement Learning
Not on the map. Why?
Abstract (arXiv)
Algorithms that exploit factored Markov decision processes are far more sample-efficient than factor-agnostic methods, yet they assume a factored representation is known a priori -- a requirement that breaks down when the agent sees only high-dimensional observations. Conversely, deep reinforcement learning handles such inputs but cannot benefit from factored structure. We address this representation problem with Action-Controllable Factorization (ACF), a contrastive learning approach that uncovers independently controllable latent variables -- state components each action can influence separately. ACF leverages sparsity: actions typically affect only a subset of variables, while the rest evolve under the environment's dynamics, yielding informative data for contrastive training. ACF recovers the ground truth controllable factors directly from pixel observations on three benchmarks with known factored structure -- Taxi, FourRooms, and MiniGrid-DoorKey -- consistently outperforming baseline disentanglement algorithms.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 65 of its 100 references that are in the map. First posted on arXiv 2 Oct 2025.
Builds on
65 of 100 references.
References · 65
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
- 2013Auto-Encoding Variational Bayescited by 3,980
- 2016Gaussian Error Linear Units (GELUs)cited by 3,431
- 2015Continuous control with deep reinforcement learningcited by 3,279
- 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
- 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
- 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 929
- 2015End-to-End Training of Deep Visuomotor Policiescited by 847
- 2018Disentangling by Factorisingcited by 614
- 2019Variational Autoencoders and Nonlinear ICA: A Unifying Frameworkcited by 494
- 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
- 2022Temporal Difference Learning for Model Predictive Controlcited by 375
38 more
- 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 320
- 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 288
- 2018Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learningcited by 278
- 2016Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICAcited by 270
- 2020Weakly-Supervised Disentanglement Without Compromisescited by 237
- 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 204
- 2021Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICAcited by 158
- 2017DARLA: Improving Zero-Shot Transfer in Reinforcement Learningcited by 147
- 2022Interventional Causal Representation Learningcited by 140
- 2022CITRIS: Causal Identifiability from Temporal Intervened Sequencescited by 132
- 2020Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Codingcited by 121
- 2023Nonparametric Identifiability of Causal Representations from Unknown Interventionscited by 105
- 2020ICE-BeeM: Identifiable Conditional Energy-Based Deep Models Based on Nonlinear ICAcited by 102
- 2020Counterfactual Data Augmentation using Locally Factored Dynamicscited by 93
- 2023Learning Linear Causal Representations from Interventions under General Nonlinear Mixingcited by 92
- 2022Linear Causal Disentanglement via Interventionscited by 84
- 2023Identifiability Guarantees for Causal Disentanglement from Soft Interventionscited by 84
- 2021Efficient Neural Causal Discovery without Acyclicity Constraintscited by 82
- 2021Causal Influence Detection for Improving Efficiency in Reinforcement Learningcited by 70
- 2022Denoised MDPs: Learning World Models Better Than the World Itselfcited by 66
- 2022Causal Dynamics Learning for Task-Independent State Abstractioncited by 64
- 2020Predictive Information Accelerates Learning in RLcited by 56
- 2021Temporal Predictive Coding For Model-Based Planning In Latent Spacecited by 54
- 2021Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learningcited by 53
- 2022Causal Representation Learning for Instantaneous and Temporal Effects in Interactive Systemscited by 50
- 2023Nonlinear Independent Component Analysis for Principled Disentanglement in Unsupervised Deep Learningcited by 48
- 2022MoCoDA: Model-based Counterfactual Data Augmentationcited by 44
- 2021Learning Markov State Abstractions for Deep Reinforcement Learningcited by 42
- 2022Guaranteed Discovery of Control-Endogenous Latent States with Multi-Step Inverse Modelscited by 42
- 2014Near-optimal Reinforcement Learning in Factored MDPscited by 41
- 2023Disentanglement via Latent Quantizationcited by 40
- 2024Nonparametric Partial Disentanglement via Mechanism Sparsity: Sparse Actions, Interventions and Sparse Temporal Dependenciescited by 39
- 2023BISCUIT: Causal Representation Learning from Binary Interactionscited by 38
- 2023General Identifiability and Achievability for Causal Representation Learningcited by 34
- 2023Conditional Mutual Information for Disentangled Representations in Reinforcement Learningcited by 27
- 2022DCI-ES: An Extended Disentanglement Framework with Connections to Identifiabilitycited by 23
- 2023Learning World Models with Identifiable Factorizationcited by 23
- 2022Temporal Disentanglement of Representations for Improved Generalisation in Reinforcement Learningcited by 19
50 most cited of 65 All 65 (JSON)
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.