mlsanityData to 7 Oct 2026

Unsupervised Visual Attention and Invariance for Reinforcement Learning

arXiv 2104.02921 ↗First posted 7 Apr 2021cs.RO cs.CVcited by 30

Not on the map. Why?

Abstract (arXiv)

Vision-based reinforcement learning (RL) is successful, but how to generalize it to unknown test environments remains challenging. Existing methods focus on training an RL policy that is universal to changing visual domains, whereas we focus on extracting visual foreground that is universal, feeding clean invariant vision to the RL policy learner. Our method is completely unsupervised, without manual annotations or access to environment internals. Given videos of actions in a training environment, we learn how to extract foregrounds with unsupervised keypoint detection, followed by unsupervised visual attention to automatically generate a foreground mask per video frame. We can then introduce artificial distractors and train a model to reconstruct the clean foreground mask from noisy observations. Only this learned model is needed during test to provide distraction-free visual input to the RL policy learner. Our Visual Attention and Invariance (VAI) method significantly outperforms the state-of-the-art on visual domain generalization, gaining 15 to 49% (61 to 229%) more cumulative rewards per episode on DeepMind Control (our DrawerWorld Manipulation) benchmarks. Our results demonstrate that it is not only possible to learn domain-invariant vision without any supervision, but freeing RL from visual distractions also makes the policy more focused and thus far better.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 47 of its 70 references that are in the map. First posted on arXiv 7 Apr 2021.

Builds on

47 of 70 references.

References · 47

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  6. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  7. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  8. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  9. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  10. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  11. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  12. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
35 more
  1. 2015Learning Transferable Features with Deep Adaptation Networkscited by 1,323
  2. 2017Adversarial Discriminative Domain Adaptationcited by 1,309
  3. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  4. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  5. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  6. 2018DeepMind Control Suitecited by 935
  7. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  8. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  9. 2018Learning Dexterous In-Hand Manipulationcited by 643
  10. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  11. 2017Learning multiple visual domains with residual adapterscited by 553
  12. 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 511
  13. 2017Robust Adversarial Reinforcement Learningcited by 439
  14. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  15. 2013Direct and Indirect Effectscited by 276
  16. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  17. 2020Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effectcited by 256
  18. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  19. 2016CAD2RL: Real Single-Image Flight without a Single Real Imagecited by 227
  20. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 220
  21. 2015Learning Multi-Domain Convolutional Neural Networks for Visual Trackingcited by 209
  22. 2017Colorization as a Proxy Task for Visual Understandingcited by 175
  23. 2016Deep Reconstruction-Classification Networks for Unsupervised Domain Adaptationcited by 162
  24. 2016Deep Reinforcement Learning from Self-Play in Imperfect-Information Gamescited by 153
  25. 2016EPOpt: Learning Robust Neural Network Policies Using Model Ensemblescited by 136
  26. 2020Generalization in Reinforcement Learning by Soft Data Augmentationcited by 132
  27. 2017Preparing for the Unknown: Learning a Universal Policy with Online System Identificationcited by 129
  28. 2020Self-Supervised Policy Adaptation during Deploymentcited by 121
  29. 2018Unsupervised Learning of Object Landmarks through Conditional Image Generationcited by 113
  30. 2019Unsupervised Learning of Object Keypoints for Perception and Controlcited by 109
  31. 2017Incremental Learning Through Deep Adaptationcited by 108
  32. 2019Towards Universal Object Detection by Domain Attentioncited by 89
  33. 2017Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learningcited by 84
  34. 2015Memory-based control with recurrent neural networkscited by 77
  35. 2019Single Episode Policy Transfer in Reinforcement Learningcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 7; 2023: 7; 2024: 6; 2025: 3; 2026: 5.

30 papers.

Sort:

By area:

  1. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
  2. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 101
  3. 2022Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learningcited by 58
  4. 2023Crossway Diffusion: Improving Diffusion-based Visuomotor Policy via Self-supervised Learningcited by 37
  5. 2022Look where you look! Saliency-guided Q-networks for generalization in visual Reinforcement Learningcited by 32
  6. 2026Maximum Likelihood Reinforcement Learningcited by 29
  7. 2022Does Self-supervised Learning Really Improve Reinforcement Learning from Pixels?cited by 24
  8. 2022A Comprehensive Survey of Data Augmentation in Visual Reinforcement Learningcited by 23
  9. 20233D Implicit Transporter for Temporally Consistent Keypoint Discoverycited by 20
  10. 2022Learning Deep Sensorimotor Policies for Vision-based Autonomous Drone Racingcited by 13
  11. 2023Generalizable Visual Reinforcement Learning with Segment Anything Modelcited by 12
  12. 2024A Recipe for Unbounded Data Augmentation in Visual Reinforcement Learningcited by 12
18 more
  1. 2026R2-Dreamer: Redundancy-Reduced World Models without Decoders or Augmentationcited by 10
  2. 2022Local Feature Swapping for Generalization in Reinforcement Learningcited by 9
  3. 2023Conceptual Reinforcement Learning for Language-Conditioned Taskscited by 8
  4. 2025Disentangled World Models: Learning to Transfer Semantic Knowledge from Distracting Videos for Reinforcement Learningcited by 6
  5. 2024Understanding What Affects the Generalization Gap in Visual Reinforcement Learning: Theory and Empirical Evidencecited by 5
  6. 2024DEAR: Disentangled Environment and Agent Representations for Reinforcement Learning without Reconstructioncited by 5
  7. 2025Merging and Disentangling Views in Visual Reinforcement Learning for Robotic Manipulationcited by 2
  8. 2022AACC: Asymmetric Actor-Critic in Contextual Reinforcement Learningcited by 1
  9. 2023The Role of Diverse Replay for Generalisation in Reinforcement Learningcited by 1
  10. 2023Self-supervised Scene Text Segmentation with Object-centric Layered Representations Augmented by Text Regionscited by 1
  11. 2023OrthoNets: Orthogonal Channel Attention Networkscited by 1
  12. 2024Efficient Training of Generalizable Visuomotor Policies via Control-Aware Augmentationcited by 1
  13. 2024Training on Irrelevant States Implies Data Augmentation: Generalization in Contextual MDPscited by 1
  14. 2025How Ensembles of Distilled Policies Improve Generalisation in Reinforcement Learningcited by 1
  15. 2024Explore-Go: Leveraging Exploration for Generalisation in Deep Reinforcement Learningcited by 0
  16. 2026Domain Adaptation with Adaptive Imagination for Visual Reinforcement Learning under Limited Target Datacited by 0
  17. 2026Task-Relevant Representation Decoupling for Visual Reinforcement Learning Generalizationcited by 0
  18. 2026StarWM: Self-Supervised Trained Attention Routing for Robust World Modelscited by 0