mlsanityData to 7 Oct 2026

Transfer RL across Observation Feature Spaces via Model-Based Regularization

arXiv 2201.00248 ↗First posted 1 Jan 2022cs.LGcited by 18

Not on the map. Why?

Abstract (arXiv)

In many reinforcement learning (RL) applications, the observation space is specified by human developers and restricted by physical realizations, and may thus be subject to dramatic changes over time (e.g. increased number of observable features). However, when the observation space changes, the previous policy will likely fail due to the mismatch of input features, and another policy must be trained from scratch, which is inefficient in terms of computation and sample complexity. Following theoretical insights, we propose a novel algorithm which extracts the latent-space dynamics in the source task, and transfers the dynamics model to the target task to use as a model-based regularizer. Our algorithm works for drastic changes of observation space (e.g. from vector-based observation to image-based observation), without any inter-task mapping or any prior knowledge of the target task. Empirical results show that our algorithm significantly improves the efficiency and stability of learning in the target task.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 31 of its 45 references that are in the map. First posted on arXiv 1 Jan 2022.

Builds on

31 of 45 references.

References · 31

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  4. 2018DeepMind Control Suitecited by 935
  5. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  6. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  7. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  8. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 288
  9. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  10. 2020Transfer Learning in Deep Reinforcement Learning: A Surveycited by 259
  11. 2018Unity: A General Platform for Intelligent Agentscited by 232
  12. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
19 more
  1. 2017Emergence of Invariance and Disentanglement in Deep Representationscited by 213
  2. 2019Contrastive Learning of Structured World Modelscited by 206
  3. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 204
  4. 2021Reinforcement Learning with Prototypical Representationscited by 179
  5. 2017DARLA: Improving Zero-Shot Transfer in Reinforcement Learningcited by 147
  6. 2019Scalable methods for computing state similarity in deterministic Markov Decision Processescited by 133
  7. 2020Invariant Causal Prediction for Block MDPscited by 104
  8. 2020Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learningcited by 101
  9. 2020Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifierscited by 93
  10. 2020The Value Equivalence Principle for Model-Based Reinforcement Learningcited by 88
  11. 2017Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learningcited by 84
  12. 2020Plannable Approximations to MDP Homomorphisms: Equivariance under Actionscited by 61
  13. 2020The Value-Improvement Path: Towards Better Representations for Reinforcement Learningcited by 60
  14. 2021On The Effect of Auxiliary Tasks on Representation Dynamicscited by 56
  15. 2019A Geometric Perspective on Optimal Representations for Reinforcement Learningcited by 54
  16. 2019Reinforcement Learning for Market Making in a Multi-agent Dealer Marketcited by 35
  17. 2017Sensor Fusion for Robot Control through Deep Reinforcement Learningcited by 7
  18. 2020TempLe: Learning Template of Transitions for Sample Efficient Multi-task RLcited by 6
  19. 2019Neural Network Surgery with Setscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 4; 2024: 5; 2025: 2; 2026: 1.

18 papers.

Sort:

By area:

  1. 2022Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progresscited by 63
  2. 2023TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learningcited by 56
  3. 2024Privileged Sensing Scaffolds Reinforcement Learningcited by 18
  4. 2022Live in the Moment: Learning Dynamics Model Adapted to Evolving Policycited by 17
  5. 2024Overcoming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RLcited by 17
  6. 2022Multi-Agent Policy Transfer via Task Relationship Modelingcited by 14
  7. 2024Premier-TACO is a Few-Shot Policy Learner: Pretraining Multitask Representation via Temporal Action-Driven Contrastive Losscited by 10
  8. 2023Making Offline RL Online: Collaborative World Models for Offline Visual Reinforcement Learningcited by 7
  9. 2022Distributional Reward Estimation for Effective Multi-Agent Deep Reinforcement Learningcited by 6
  10. 2025Disentangled World Models: Learning to Transfer Semantic Knowledge from Distracting Videos for Reinforcement Learningcited by 6
  11. 2024UVIS: Unsupervised Video Instance Segmentationcited by 5
  12. 2023Online Prototype Alignment for Few-shot Policy Transfercited by 4
6 more
  1. 2022Provably Sample-Efficient RL with Side Information about Latent Dynamicscited by 2
  2. 2023Accelerating Policy Gradient by Estimating Value Function from Prior Computation in Deep Reinforcement Learningcited by 2
  3. 2024Cross-Domain Policy Transfer by Representation Alignment via Multi-Domain Behavioral Cloningcited by 1
  4. 2022Planning Immediate Landmarks of Targets for Model-Free Skill Transfer across Agentscited by 0
  5. 2025Planning Under Observation Mismatch for Traffic Signal Control via Adaptive Modular World Modelscited by 0
  6. 2026Domain Adaptation with Adaptive Imagination for Visual Reinforcement Learning under Limited Target Datacited by 0