# Multistep Inverse Is Not All You Need

arXiv: https://arxiv.org/abs/2403.11940 · HTML: https://mlsanity.com/paper/2403.11940/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 25 references that are in the map.
- First posted on arXiv 18 Mar 2024.

## Builds on (most cited first, 13 of 13)

- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Learning Invariant Representations for Reinforcement Learning without Reconstruction](https://mlsanity.com/paper/2006.10742/) (2020)
- [DeepMDP: Learning Continuous Latent Space Models for Representation Learning](https://mlsanity.com/paper/1906.02736/) (2019)
- [Provably efficient RL with Rich Observations via Latent State Decoding](https://mlsanity.com/paper/1901.09018/) (2019)
- [Kinematic State Abstraction and Provably Efficient Rich-Observation Reinforcement Learning](https://mlsanity.com/paper/1911.05815/) (2019)
- [Deep Reinforcement and InfoMax Learning](https://mlsanity.com/paper/2006.07217/) (2020)
- [Denoised MDPs: Learning World Models Better Than the World Itself](https://mlsanity.com/paper/2206.15477/) (2022)
- [Guaranteed Discovery of Control-Endogenous Latent States with Multi-Step Inverse Models](https://mlsanity.com/paper/2207.08229/) (2022)
- [Sample-Efficient Reinforcement Learning in the Presence of Exogenous Information](https://mlsanity.com/paper/2206.04282/) (2022)
- [Representation Learning with Multi-Step Inverse Kinematics: An Efficient and Optimal Approach to Rich-Observation RL](https://mlsanity.com/paper/2304.05889/) (2023)
- [Learning the Linear Quadratic Regulator from Nonlinear Observations](https://mlsanity.com/paper/2010.03799/) (2020)
- [Agent-Controller Representations: Principled Offline RL with Rich Exogenous Information](https://mlsanity.com/paper/2211.00164/) (2022)

## Built on by (most cited first, 11 of 11)

- [Latent Action Learning Requires Supervision in the Presence of Distractors](https://mlsanity.com/paper/2502.00379/) (2025)
- [CLAM: Continuous Latent Action Models for Robot Learning from Unlabeled Demonstrations](https://mlsanity.com/paper/2505.04999/) (2025)
- [Sensorimotor World Models: Perception for Action via Inverse Dynamics](https://mlsanity.com/paper/2606.20104/) (2026)
- [Segment to Focus: Guiding Latent Action Models in the Presence of Distractors](https://mlsanity.com/paper/2602.02259/) (2026)
- [Learning a Fast Mixing Exogenous Block MDP using a Single Trajectory](https://mlsanity.com/paper/2410.03016/) (2024)
- [When Does Predictive Inverse Dynamics Outperform Behavior Cloning?](https://mlsanity.com/paper/2601.21718/) (2026)
- [Offline Action-Free Learning of Ex-BMDPs by Comparing Diverse Datasets](https://mlsanity.com/paper/2503.21018/) (2025)
- [Dueling World Models: Advantage-Style Action Channels for Common-Mode Distractor Rejection](https://mlsanity.com/paper/2608.06706/) (2026)
- [Tracking control of latent dynamic systems with application to spacecraft attitude control](https://mlsanity.com/paper/2412.06342/) (2024)
- [What a World Model Represents Is Three Questions](https://mlsanity.com/paper/2607.06640/) (2026)
- [Keeping JEPA World Models Plannable When Little of the Frame Moves](https://mlsanity.com/paper/2610.03137/) (2026)

## Abstract (arXiv)

In real-world control settings, the observation space is often unnecessarily high-dimensional and subject to time-correlated noise. However, the controllable dynamics of the system are often far simpler than the dynamics of the raw observations. It is therefore desirable to learn an encoder to map the observation space to a simpler space of control-relevant variables. In this work, we consider the Ex-BMDP model, first proposed by Efroni et al. (2022), which formalizes control problems where observations can be factorized into an action-dependent latent state which evolves deterministically, and action-independent time-correlated noise. Lamb et al. (2022) proposes the "AC-State" method for learning an encoder to extract a complete action-dependent latent state representation from the observations in such problems. AC-State is a multistep-inverse method, in that it uses the encoding of the the first and last state in a path to predict the first action in the path. However, we identify cases where AC-State will fail to learn a correct latent representation of the agent-controllable factor of the state. We therefore propose a new algorithm, ACDF, which combines multistep-inverse prediction with a latent forward model. ACDF is guaranteed to correctly infer an action-dependent latent state encoder for a large class of Ex-BMDP models. We demonstrate the effectiveness of ACDF on tabular Ex-BMDPs through numerical simulations; as well as high-dimensional environments using neural-network-based encoders. Code is available at https://github.com/midi-lab/acdf.