mlsanityData to 14 Oct 2026

Agent-Controller Representations: Principled Offline RL with Rich Exogenous Information

arXiv 2211.00164 ↗First posted 31 Oct 2022cs.LG cs.AI cs.CVcited by 16

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Learning to control an agent from data collected offline in a rich pixel-based visual observation space is vital for real-world applications of reinforcement learning (RL). A major challenge in this setting is the presence of input information that is hard to model and irrelevant to controlling the agent. This problem has been approached by the theoretical RL community through the lens of exogenous information, i.e, any control-irrelevant information contained in observations. For example, a robot navigating in busy streets needs to ignore irrelevant information, such as other people walking in the background, textures of objects, or birds in the sky. In this paper, we focus on the setting with visually detailed exogenous information, and introduce new offline RL benchmarks offering the ability to study this problem. We find that contemporary representation learning techniques can fail on datasets where the noise is a complex and time dependent process, which is prevalent in practical applications. To address these, we propose to use multi-step inverse models, which have seen a great deal of interest in the RL theory community, to learn Agent-Controller Representations for Offline-RL (ACRO). Despite being simple and requiring no reward, we show theoretically and empirically that the representation created by this objective greatly outperforms baselines.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 44 of its 58 references that are in the map. First posted on arXiv 31 Oct 2022.

Builds on

44 of 58 references.

Closest ancestors · 11

Same direction, newest first

  1. 2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations9 Jun
  2. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation21 Nov
  3. 2021Representation Learning for Online and Offline RL in Low-rank MDPs9 Oct
  4. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  5. 2021BRAC+: Improved Behavior Regularized Actor Critic for Offline Reinforcement Learning2 Oct
  6. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  7. 2021Representation Matters: Offline Pretraining for Sequential Decision Making11 Feb
  8. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  9. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 33

  1. 2017Neural Discrete Representation Learningcited by 4,712
  2. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  5. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  6. 2018DeepMind Control Suitecited by 956
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  8. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  9. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  10. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
  11. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 318
  12. 2019Causal Confusion in Imitation Learningcited by 288
21 more
  1. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  2. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  3. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 230
  4. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 144
  5. 2019Kinematic State Abstraction and Provably Efficient Rich-Observation Reinforcement Learningcited by 114
  6. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
  7. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  8. 2020Deep Reinforcement and InfoMax Learningcited by 79
  9. 2022BYOL-Explore: Exploration by Bootstrapped Predictioncited by 73
  10. 2022Denoised MDPs: Learning World Models Better Than the World Itselfcited by 68
  11. 2022Causal Dynamics Learning for Task-Independent State Abstractioncited by 65
  12. 2020Planning from Pixels using Inverse Dynamics Modelscited by 36
  13. 2021Provable Representation Learning for Imitation with Contrastive Fourier Featurescited by 34
  14. 2020Representations for Stable Off-Policy Reinforcement Learningcited by 31
  15. 2022Sample-Efficient Reinforcement Learning in the Presence of Exogenous Informationcited by 26
  16. 2018Discovering and Removing Exogenous State Variables and Rewards for Reinforcement Learningcited by 24
  17. 2021Learning Representations for Pixel-based Control: What Matters and Why?cited by 24
  18. 2019Safe Policy Improvement with Soft Baseline Bootstrappingcited by 22
  19. 2022An Empirical Investigation of Representation Learning for Imitationcited by 15
  20. 2021Sparsity in Partially Controllable Linear Systemscited by 9
  21. 2022Uniqueness and Complexity of Inverse MDP Modelscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 3; 2026: 7.

16 papers.

Sort:

By area:

  1. 2024Learning Latent Dynamic Robust Representations for World Modelscited by 21
  2. 2024Model-based Offline Reinforcement Learning with Lower Expectile Q-Learningcited by 17
  3. 2025A Survey of State Representation Learning for Deep Reinforcement Learningcited by 17
  4. 2024Multistep Inverse Is Not All You Needcited by 11
  5. 2023Reinforcement Learning with Exogenous States and Rewardscited by 4
  6. 2024Generalizing Multi-Step Inverse Models for Representation Learning to Finite-Memory POMDPscited by 2
  7. 2025Offline Action-Free Learning of Ex-BMDPs by Comparing Diverse Datasetscited by 2
  8. 2025PvP: Data-Efficient Humanoid Robot Learning with Proprioceptive-Privileged Contrastive Representationscited by 2
  9. 2026Dueling World Models: Advantage-Style Action Channels for Common-Mode Distractor Rejectioncited by 2
  10. 2026Action-Sufficient Goal Representationscited by 1
  11. 2024Stable Offline Value Function Learning with Bisimulation-based Representationscited by 0
  12. 2026Gradient-Momentum Coupling: A Parameter-Space Proxy for Learning Progresscited by 0
4 more
  1. 2026Exploiting Local Dynamics Regularity for Reusable Skills in Offline Hierarchical RLcited by 0
  2. 2026FlowMo-WM: A World Model with Object Momentum and Hidden Ambient Driftcited by 0
  3. 2026Mastering Atari 2600 Games with Discovered Optionscited by 0
  4. 2026CausalDreamer: Learning Predictive World Models with Latent Disentanglementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020Offline Reinforcement Learning from Images with Latent Space Models
  2. 2025Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models
  3. 2024DEAR: Disentangled Environment and Agent Representations for Reinforcement Learning without Reconstruction
  4. 2025Video-Enhanced Offline Reinforcement Learning: A Model-Based Approach
  5. 2023Making Offline RL Online: Collaborative World Models for Offline Visual Reinforcement Learning
  6. 2024The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learning
  7. 2022Behavior Prior Representation learning for Offline Reinforcement Learning
  8. 2024Offline Actor-Critic Reinforcement Learning Scales to Large Models