mlsanityData to 7 Oct 2026

Guaranteed Discovery of Control-Endogenous Latent States with Multi-Step Inverse Models

arXiv 2207.08229 ↗First posted 17 Jul 2022cs.LG cs.RO stat.MLcited by 42

Not on the map. Why?

Abstract (arXiv)

In many sequential decision-making tasks, the agent is not able to model the full complexity of the world, which consists of multitudes of relevant and irrelevant information. For example, a person walking along a city street who tries to model all aspects of the world would quickly be overwhelmed by a multitude of shops, cars, and people moving in and out of view, each following their own complex and inscrutable dynamics. Is it possible to turn the agent's firehose of sensory information into a minimal latent state that is both necessary and sufficient for an agent to successfully act in the world? We formulate this question concretely, and propose the Agent Control-Endogenous State Discovery algorithm (AC-State), which has theoretical guarantees and is practically demonstrated to discover the minimal control-endogenous latent state which contains all of the information necessary for controlling the agent, while fully discarding all irrelevant information. This algorithm consists of a multi-step inverse model (predicting actions from distant observations) with an information bottleneck. AC-State enables localization, exploration, and navigation without reward or demonstrations. We demonstrate the discovery of the control-endogenous latent state in three domains: localizing a robot arm with distractions (e.g., changing lighting conditions and background), exploring a maze alongside other agents, and navigating in the Matterport house simulator.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 40 of its 69 references that are in the map. First posted on arXiv 17 Jul 2022.

Builds on

40 of 69 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2017Neural Discrete Representation Learningcited by 4,650
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  7. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  8. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  9. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  10. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  11. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  12. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
28 more
  1. 2016Deep Variational Information Bottleneckcited by 915
  2. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  3. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  4. 2016Past, Present, and Future of Simultaneous Localization And Mapping: Towards the Robust-Perception Agecited by 405
  5. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  6. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  7. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
  8. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 266
  9. 2021Mastering Atari Games with Limited Datacited by 235
  10. 2015Incentivizing Exploration In Reinforcement Learning With Deep Predictive Modelscited by 154
  11. 2015Variational Information Maximisation for Intrinsically Motivated Reinforcement Learningcited by 142
  12. 2019Kinematic State Abstraction and Provably Efficient Rich-Observation Reinforcement Learningcited by 114
  13. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
  14. 2020Deep Reinforcement and InfoMax Learningcited by 79
  15. 2020Causal Discovery in Physical Systems from Videoscited by 77
  16. 2022BYOL-Explore: Exploration by Bootstrapped Predictioncited by 71
  17. 2022Denoised MDPs: Learning World Models Better Than the World Itselfcited by 66
  18. 2022Causal Dynamics Learning for Task-Independent State Abstractioncited by 64
  19. 2021Learning Task Informed Abstractionscited by 58
  20. 2015From Pixels to Torques: Policy Learning with Deep Dynamical Modelscited by 44
  21. 2021Learning Neural Causal Models with Active Interventionscited by 40
  22. 2021Which Mutual-Information Representation Learning Objectives are Sufficient for Control?cited by 28
  23. 2022INFOrmation Prioritization through EmPOWERment in Visual Model-Based RLcited by 27
  24. 2022Sample-Efficient Reinforcement Learning in the Presence of Exogenous Informationcited by 26
  25. 2018Discovering and Removing Exogenous State Variables and Rewards for Reinforcement Learningcited by 24
  26. 2019Unsupervised Visuomotor Control through Distributional Planning Networkscited by 9
  27. 2021Sparsity in Partially Controllable Linear Systemscited by 9
  28. 2022Uniqueness and Complexity of Inverse MDP Modelscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 5; 2024: 12; 2025: 10; 2026: 14.

42 papers.

Sort:

By area:

  1. 2024Foundation Policies with Hilbert Representationscited by 89
  2. 2023TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learningcited by 56
  3. 2025Latent Action Learning Requires Supervision in the Presence of Distractorscited by 40
  4. 2024Learning Latent Dynamic Robust Representations for World Modelscited by 20
  5. 2025A Survey of State Representation Learning for Deep Reinforcement Learningcited by 17
  6. 2023Mastering Robot Manipulation with Multimodal Prompts through Pretraining and Multi-task Fine-tuningcited by 15
  7. 2026Sensorimotor World Models: Perception for Action via Inverse Dynamicscited by 14
  8. 2025Dual Goal Representationscited by 13
  9. 2023Guide Your Agent with Adaptive Multimodal Rewardscited by 10
  10. 2024Multistep Inverse Is Not All You Needcited by 10
  11. 2024Learning Cognitive Maps from Transformer Representations for Efficient Planning in Partially Observed Environmentscited by 6
  12. 2024Policy-shaped prediction: avoiding distractions in model-based reinforcement learningcited by 6
30 more
  1. 2024Learning Action-based Representations Using Invariancecited by 5
  2. 2024DMC-VB: A Benchmark for Representation Learning for Control with Visual Distractorscited by 5
  3. 2025Adapting a World Model for Trajectory Following in a 3D Gamecited by 5
  4. 2026Segment to Focus: Guiding Latent Action Models in the Presence of Distractorscited by 5
  5. 2023Reinforcement Learning with Exogenous States and Rewardscited by 4
  6. 2024Learning Abstract World Model for Value-preserving Planning with Optionscited by 4
  7. 2026No Gaussian Required: Contrastive Inverse Dynamics for JEPA World Modelscited by 4
  8. 2022Cost-Driven Representation Learning for Linear Quadratic Gaussian Control: Part Icited by 3
  9. 2024Learning a Fast Mixing Exogenous Block MDP using a Single Trajectorycited by 3
  10. 2026When Does Predictive Inverse Dynamics Outperform Behavior Cloning?cited by 3
  11. 2024Generalizing Multi-Step Inverse Models for Representation Learning to Finite-Memory POMDPscited by 2
  12. 2025Offline Action-Free Learning of Ex-BMDPs by Comparing Diverse Datasetscited by 2
  13. 2025From Pixels to Factors: Learning Independently Controllable State Variables for Reinforcement Learningcited by 2
  14. 2025PvP: Data-Efficient Humanoid Robot Learning with Proprioceptive-Privileged Contrastive Representationscited by 2
  15. 2024Reward-Relevance-Filtered Linear Offline Reinforcement Learningcited by 1
  16. 2024SeMOPO: Learning High-quality Model and Policy from Low-quality Offline Visual Datasetscited by 1
  17. 2025Learning Fused State Representations for Control from Multi-View Observationscited by 1
  18. 2026Action-Sufficient Goal Representationscited by 1
  19. 2026Why Latent Actions Fail, and How to Prevent Itcited by 1
  20. 2026Compositional Transduction with Latent Analogies for Offline Goal-Conditioned Reinforcement Learningcited by 1
  21. 2023Fast exploration and learning of latent graphs with aliased observationscited by 0
  22. 2025How Well Do Latent World Models Understand Partially Observable Safety Constraints?cited by 0
  23. 2025Reinforcement Learning and Consumption-Savings Behaviorcited by 0
  24. 2026Learning to Perceive the World Through Control: Empowerment-Based Representation Learningcited by 0
  25. 2026What a World Model Represents Is Three Questionscited by 0
  26. 2026Latent Lie-Poisson Neural Networks (LLPNNs): Discovering the motion of Lie-Poisson systems through observable data and latent dynamicscited by 0
  27. 2026TaskSense: Focusing on What Matters in World Modelscited by 0
  28. 2026Dueling World Models: Advantage-Style Action Channels for Common-Mode Distractor Rejectioncited by 0
  29. 2026Minimal Recurrent Behavioral Memory for Imitation under Partial Observabilitycited by 0
  30. 2026CausalDreamer: Learning Predictive World Models with Latent Disentanglementcited by 0