Generalizing Multi-Step Inverse Models for Representation Learning to Finite-Memory POMDPs
Not on the map. Why?
Abstract (arXiv)
Discovering an informative, or agent-centric, state representation that encodes only the relevant information while discarding the irrelevant is a key challenge towards scaling reinforcement learning algorithms and efficiently applying them to downstream tasks. Prior works studied this problem in high-dimensional Markovian environments, when the current observation may be a complex object but is sufficient to decode the informative state. In this work, we consider the problem of discovering the agent-centric state in the more challenging high-dimensional non-Markovian setting, when the state can be decoded from a sequence of past observations. We establish that generalized inverse models can be adapted for learning agent-centric state representation for this task. Our results include asymptotic theory in the deterministic dynamics setting as well as counter-examples for alternative intuitive algorithms. We complement these findings with a thorough empirical study on the agent-centric state discovery abilities of the different alternatives we put forward. Particularly notable is our analysis of past actions, where we show that these can be a double-edged sword: making the algorithms more successful when used correctly and causing dramatic failure when used incorrectly.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 40 of its 49 references that are in the map. First posted on arXiv 22 Apr 2024.
Builds on
40 of 49 references.
References · 40
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
- 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
- 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
- 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
- 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
- 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
- 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
- 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
- 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
- 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
- 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
28 more
- 2019Causal Confusion in Imitation Learningcited by 286
- 2017Neural Episodic Controlcited by 148
- 2018Deep Variational Reinforcement Learning for POMDPscited by 122
- 2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 115
- 2019Kinematic State Abstraction and Provably Efficient Rich-Observation Reinforcement Learningcited by 114
- 2018Near-Optimal Representation Learning for Hierarchical Reinforcement Learningcited by 107
- 2022When Is Partially Observable Reinforcement Learning Not Scary?cited by 100
- 2020Deep Reinforcement and InfoMax Learningcited by 79
- 2016Loss is its own Reward: Self-Supervision for Reinforcement Learningcited by 72
- 2022BYOL-Explore: Exploration by Bootstrapped Predictioncited by 71
- 2022Denoised MDPs: Learning World Models Better Than the World Itselfcited by 66
- 2022Causal Dynamics Learning for Task-Independent State Abstractioncited by 64
- 2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observationscited by 63
- 2022PAC Reinforcement Learning for Predictive State Representationscited by 42
- 2022Guaranteed Discovery of Control-Endogenous Latent States with Multi-Step Inverse Modelscited by 42
- 2019Learning Predictive Models From Observation and Interactioncited by 41
- 2022Provable Reinforcement Learning with a Short-Term Memorycited by 40
- 2022Understanding Self-Predictive Learning for Reinforcement Learningcited by 40
- 2023Inverse Dynamics Pretraining Learns Good Representations for Multitask Imitationcited by 30
- 2021Which Mutual-Information Representation Learning Objectives are Sufficient for Control?cited by 28
- 2022INFOrmation Prioritization through EmPOWERment in Visual Model-Based RLcited by 27
- 2022Sample-Efficient Reinforcement Learning in the Presence of Exogenous Informationcited by 26
- 2018Discovering and Removing Exogenous State Variables and Rewards for Reinforcement Learningcited by 24
- 2023Representation Learning with Multi-Step Inverse Kinematics: An Efficient and Optimal Approach to Rich-Observation RLcited by 22
- 2022Embed to Control Partially Observed Systems: Representation Learning with Provable Sample Efficiencycited by 16
- 2022Agent-Controller Representations: Principled Offline RL with Rich Exogenous Informationcited by 15
- 2021Sparsity in Partially Controllable Linear Systemscited by 9
- 2023Enhancing Treatment Effect Estimation: A Model Robust Approach Integrating Randomized Experiments and External Controls using the Double Penalty Integration Estimatorcited by 6
Built on by
Citing papers per year: 2024: 1; 2025: 0; 2026: 1.
2 papers.