mlsanityData to 7 Oct 2026

Temporal Disentanglement of Representations for Improved Generalisation in Reinforcement Learning

arXiv 2207.05480 ↗First posted 12 Jul 2022cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

Reinforcement Learning (RL) agents are often unable to generalise well to environment variations in the state space that were not observed during training. This issue is especially problematic for image-based RL, where a change in just one variable, such as the background colour, can change many pixels in the image. The changed pixels can lead to drastic changes in the agent's latent representation of the image, causing the learned policy to fail. To learn more robust representations, we introduce TEmporal Disentanglement (TED), a self-supervised auxiliary task that leads to disentangled image representations exploiting the sequential nature of RL observations. We find empirically that RL algorithms utilising TED as an auxiliary task adapt more quickly to changes in environment variables with continued training compared to state-of-the-art representation learning methods. Since TED enforces a disentangled structure of the representation, our experiments also show that policies trained with TED generalise better to unseen values of variables irrelevant to the task (e.g. background colour) as well as unseen values of variables that affect the optimal policy (e.g. goal positions).

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 38 of its 46 references that are in the map. First posted on arXiv 12 Jul 2022.

Builds on

38 of 46 references.

References · 38

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2013Auto-Encoding Variational Bayescited by 3,980
  6. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 929
  7. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  8. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  9. 2018Disentangling by Factorisingcited by 614
  10. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  11. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  12. 2020Reinforcement Learning with Augmented Datacited by 452
26 more
  1. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  2. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  3. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  4. 2018Quantifying Generalization in Reinforcement Learningcited by 345
  5. 2018Understanding disentangling in β-VAEcited by 332
  6. 2016Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICAcited by 270
  7. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  8. 2020Weakly-Supervised Disentanglement Without Compromisescited by 237
  9. 2018Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experiencecited by 203
  10. 2018A Study on Overfitting in Deep Reinforcement Learningcited by 170
  11. 2017DARLA: Improving Zero-Shot Transfer in Reinforcement Learningcited by 147
  12. 2020Generalization in Reinforcement Learning by Soft Data Augmentationcited by 132
  13. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
  14. 2020Self-Supervised Policy Adaptation during Deploymentcited by 121
  15. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
  16. 2020Invariant Causal Prediction for Block MDPscited by 104
  17. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 101
  18. 2021The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixelscited by 97
  19. 2019Weakly Supervised Disentanglement with Guaranteescited by 82
  20. 2020Deep Reinforcement and InfoMax Learningcited by 79
  21. 2021panda-gym: Open-source goal-conditioned environments for robotic learningcited by 56
  22. 2020S3K: Self-Supervised Semantic Keypoints for Robotic Manipulation via Multi-View Consistencycited by 28
  23. 2022On the Generalization of Representations in Reinforcement Learningcited by 28
  24. 2021Domain Adversarial Reinforcement Learningcited by 19
  25. 2019Affordance Learning for End-to-End Visuomotor Robot Controlcited by 18
  26. 2017From the Lab to the Desert: Fast Prototyping and Learning of Robot Locomotioncited by 3

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 3; 2024: 7; 2025: 8.

19 papers.

Sort:

By area:

  1. 2023Conditional Mutual Information for Disentangled Representations in Reinforcement Learningcited by 27
  2. 2025A Survey of State Representation Learning for Deep Reinforcement Learningcited by 17
  3. 2024Multi-view Disentanglement for Reinforcement Learning with Multiple Camerascited by 8
  4. 2025Studying the Interplay Between the Actor and Critic Representations in Reinforcement Learningcited by 7
  5. 2024Learning to Play Atari in a World of Tokenscited by 6
  6. 2025Disentangled World Models: Learning to Transfer Semantic Knowledge from Distracting Videos for Reinforcement Learningcited by 6
  7. 2024DEAR: Disentangled Environment and Agent Representations for Reinforcement Learning without Reconstructioncited by 5
  8. 2025Enhancing Tactile-based Reinforcement Learning for Robotic Controlcited by 5
  9. 2022Deep Reinforcement Learning for Multi-Agent Interactioncited by 3
  10. 2023Learning Generalizable Agents via Saliency-Guided Features Decorrelationcited by 3
  11. 2024Zero-Shot Generalization of Vision-Based RL Without Data Augmentationcited by 3
  12. 2025From Pixels to Factors: Learning Independently Controllable State Variables for Reinforcement Learningcited by 2
7 more
  1. 2025PvP: Data-Efficient Humanoid Robot Learning with Proprioceptive-Privileged Contrastive Representationscited by 2
  2. 2023Contextual Pre-planning on Reward Machine Abstractions for Enhanced Transfer in Deep Reinforcement Learningcited by 1
  3. 2024Entropy Regularized Task Representation Learning for Offline Meta-Reinforcement Learningcited by 1
  4. 2025Salience-Invariant Consistent Policy Learning for Generalization in Visual Reinforcement Learningcited by 1
  5. 2024Highway Graph to Accelerate Reinforcement Learningcited by 0
  6. 2024Skill-aware Mutual Information Optimisation for Generalisation in Reinforcement Learningcited by 0
  7. 2025MInCo: Mitigating Information Conflicts in Distracted Visual Model-based Reinforcement Learningcited by 0