# Guaranteed Discovery of Control-Endogenous Latent States with Multi-Step Inverse Models

arXiv: https://arxiv.org/abs/2207.08229 · HTML: https://mlsanity.com/paper/2207.08229/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 69 references that are in the map.
- First posted on arXiv 17 Jul 2022.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [Learning Invariant Representations for Reinforcement Learning without Reconstruction](https://mlsanity.com/paper/2006.10742/) (2020)
- [Past, Present, and Future of Simultaneous Localization And Mapping: Towards the Robust-Perception Age](https://mlsanity.com/paper/1606.05830/) (2016)
- [Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos](https://mlsanity.com/paper/2206.11795/) (2022)
- [Data-Efficient Reinforcement Learning with Self-Predictive Representations](https://mlsanity.com/paper/2007.05929/) (2020)
- [Agent57: Outperforming the Atari Human Benchmark](https://mlsanity.com/paper/2003.13350/) (2020)
- [Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images](https://mlsanity.com/paper/1506.07365/) (2015)
- [Mastering Atari Games with Limited Data](https://mlsanity.com/paper/2111.00210/) (2021)
- [Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models](https://mlsanity.com/paper/1507.00814/) (2015)
- [Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning](https://mlsanity.com/paper/1509.08731/) (2015)
- [Kinematic State Abstraction and Provably Efficient Rich-Observation Reinforcement Learning](https://mlsanity.com/paper/1911.05815/) (2019)
- [Pretraining Representations for Data-Efficient Reinforcement Learning](https://mlsanity.com/paper/2106.04799/) (2021)
- [Deep Reinforcement and InfoMax Learning](https://mlsanity.com/paper/2006.07217/) (2020)
- [Causal Discovery in Physical Systems from Videos](https://mlsanity.com/paper/2007.00631/) (2020)
- [BYOL-Explore: Exploration by Bootstrapped Prediction](https://mlsanity.com/paper/2206.08332/) (2022)
- [Denoised MDPs: Learning World Models Better Than the World Itself](https://mlsanity.com/paper/2206.15477/) (2022)
- [Causal Dynamics Learning for Task-Independent State Abstraction](https://mlsanity.com/paper/2206.13452/) (2022)

## Built on by (most cited first, 30 of 45)

- [Foundation Policies with Hilbert Representations](https://mlsanity.com/paper/2402.15567/) (2024)
- [TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learning](https://mlsanity.com/paper/2306.13229/) (2023)
- [Latent Action Learning Requires Supervision in the Presence of Distractors](https://mlsanity.com/paper/2502.00379/) (2025)
- [Learning Latent Dynamic Robust Representations for World Models](https://mlsanity.com/paper/2405.06263/) (2024)
- [Sensorimotor World Models: Perception for Action via Inverse Dynamics](https://mlsanity.com/paper/2606.20104/) (2026)
- [A Survey of State Representation Learning for Deep Reinforcement Learning](https://mlsanity.com/paper/2506.17518/) (2025)
- [Dual Goal Representations](https://mlsanity.com/paper/2510.06714/) (2025)
- [Mastering Robot Manipulation with Multimodal Prompts through Pretraining and Multi-task Fine-tuning](https://mlsanity.com/paper/2310.09676/) (2023)
- [Multistep Inverse Is Not All You Need](https://mlsanity.com/paper/2403.11940/) (2024)
- [Guide Your Agent with Adaptive Multimodal Rewards](https://mlsanity.com/paper/2309.10790/) (2023)
- [Learning Action-based Representations Using Invariance](https://mlsanity.com/paper/2403.16369/) (2024)
- [Learning Cognitive Maps from Transformer Representations for Efficient Planning in Partially Observed Environments](https://mlsanity.com/paper/2401.05946/) (2024)
- [Policy-shaped prediction: avoiding distractions in model-based reinforcement learning](https://mlsanity.com/paper/2412.05766/) (2024)
- [Segment to Focus: Guiding Latent Action Models in the Presence of Distractors](https://mlsanity.com/paper/2602.02259/) (2026)
- [DMC-VB: A Benchmark for Representation Learning for Control with Visual Distractors](https://mlsanity.com/paper/2409.18330/) (2024)
- [Adapting a World Model for Trajectory Following in a 3D Game](https://mlsanity.com/paper/2504.12299/) (2025)
- [No Gaussian Required: Contrastive Inverse Dynamics for JEPA World Models](https://mlsanity.com/paper/2608.17542/) (2026)
- [Reinforcement Learning with Exogenous States and Rewards](https://mlsanity.com/paper/2303.12957/) (2023)
- [Learning Abstract World Model for Value-preserving Planning with Options](https://mlsanity.com/paper/2406.15850/) (2024)
- [Cost-Driven Representation Learning for Linear Quadratic Gaussian Control: Part I](https://mlsanity.com/paper/2212.14511/) (2022)
- [Learning a Fast Mixing Exogenous Block MDP using a Single Trajectory](https://mlsanity.com/paper/2410.03016/) (2024)
- [From Pixels to Factors: Learning Independently Controllable State Variables for Reinforcement Learning](https://mlsanity.com/paper/2510.02484/) (2025)
- [When Does Predictive Inverse Dynamics Outperform Behavior Cloning?](https://mlsanity.com/paper/2601.21718/) (2026)
- [Generalizing Multi-Step Inverse Models for Representation Learning to Finite-Memory POMDPs](https://mlsanity.com/paper/2404.14552/) (2024)
- [Offline Action-Free Learning of Ex-BMDPs by Comparing Diverse Datasets](https://mlsanity.com/paper/2503.21018/) (2025)
- [PvP: Data-Efficient Humanoid Robot Learning with Proprioceptive-Privileged Contrastive Representations](https://mlsanity.com/paper/2512.13093/) (2025)
- [Why Latent Actions Fail, and How to Prevent It](https://mlsanity.com/paper/2605.20223/) (2026)
- [Minimal Markovization via Stable Quotients in Holonomy-Cover Decision Processes](https://mlsanity.com/paper/2607.27132/) (2026)
- [Dueling World Models: Advantage-Style Action Channels for Common-Mode Distractor Rejection](https://mlsanity.com/paper/2608.06706/) (2026)
- [Reward-Relevance-Filtered Linear Offline Reinforcement Learning](https://mlsanity.com/paper/2401.12934/) (2024)

## Abstract (arXiv)

In many sequential decision-making tasks, the agent is not able to model the full complexity of the world, which consists of multitudes of relevant and irrelevant information. For example, a person walking along a city street who tries to model all aspects of the world would quickly be overwhelmed by a multitude of shops, cars, and people moving in and out of view, each following their own complex and inscrutable dynamics. Is it possible to turn the agent's firehose of sensory information into a minimal latent state that is both necessary and sufficient for an agent to successfully act in the world? We formulate this question concretely, and propose the Agent Control-Endogenous State Discovery algorithm (AC-State), which has theoretical guarantees and is practically demonstrated to discover the minimal control-endogenous latent state which contains all of the information necessary for controlling the agent, while fully discarding all irrelevant information. This algorithm consists of a multi-step inverse model (predicting actions from distant observations) with an information bottleneck. AC-State enables localization, exploration, and navigation without reward or demonstrations. We demonstrate the discovery of the control-endogenous latent state in three domains: localizing a robot arm with distractions (e.g., changing lighting conditions and background), exploring a maze alongside other agents, and navigating in the Matterport house simulator.