mlsanityData to 7 Oct 2026

Look where you look! Saliency-guided Q-networks for generalization in visual Reinforcement Learning

arXiv 2209.09203 ↗First posted 16 Sep 2022cs.LG cs.AI cs.CVcited by 32

Not on the map. Why?

Abstract (arXiv)

Deep reinforcement learning policies, despite their outstanding efficiency in simulated visual control tasks, have shown disappointing ability to generalize across disturbances in the input training images. Changes in image statistics or distracting background elements are pitfalls that prevent generalization and real-world applicability of such control policies. We elaborate on the intuition that a good visual policy should be able to identify which pixels are important for its decision, and preserve this identification of important sources of information across images. This implies that training of a policy with small generalization gap should focus on such important pixels and ignore the others. This leads to the introduction of saliency-guided Q-networks (SGQN), a generic method for visual reinforcement learning, that is compatible with any value function learning method. SGQN vastly improves the generalization capability of Soft Actor-Critic agents and outperforms existing stateof-the-art methods on the Deepmind Control Generalization benchmark, setting a new reference in terms of training efficiency, generalization gap, and policy interpretability.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 56 of its 71 references that are in the map. First posted on arXiv 16 Sep 2022.

Builds on

56 of 71 references.

References · 56

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  3. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  6. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  7. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  8. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  9. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  10. 2017SmoothGrad: removing noise by adding noisecited by 1,097
  11. 2014Striving for Simplicity: The All Convolutional Netcited by 983
  12. 2017Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networkscited by 936
38 more
  1. 2018RISE: Randomized Input Sampling for Explanation of Black-box Modelscited by 825
  2. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  3. 2017Deep Reinforcement Learning that Matterscited by 727
  4. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  5. 2021Domain Generalization with MixStylecited by 563
  6. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  7. 2020Reinforcement Learning with Augmented Datacited by 452
  8. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  9. 2018Quantifying Generalization in Reinforcement Learningcited by 345
  10. 2019A Theory of Regularized Markov Decision Processescited by 279
  11. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  12. 2020Prioritized Level Replaycited by 190
  13. 2018A Study on Overfitting in Deep Reinforcement Learningcited by 170
  14. 2017DARLA: Improving Zero-Shot Transfer in Reinforcement Learningcited by 147
  15. 2017Visualizing and Understanding Atari Agentscited by 141
  16. 2018Assessing Generalization in Deep Reinforcement Learningcited by 132
  17. 2020Generalization in Reinforcement Learning by Soft Data Augmentationcited by 132
  18. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
  19. 2018Generalization and Regularization in DQNcited by 117
  20. 2019Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learningcited by 102
  21. 2019Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneckcited by 101
  22. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 101
  23. 2021The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixelscited by 97
  24. 2019Observational Overfitting in Reinforcement Learningcited by 90
  25. 2018A Dissection of Overfitting and Generalization in Continuous Reinforcement Learningcited by 86
  26. 2020Improving Generalization in Reinforcement Learning with Mixture Regularizationcited by 81
  27. 2021Decoupling Value and Policy for Generalization in Reinforcement Learningcited by 81
  28. 2021Improving Deep Learning Interpretability by Saliency Guided Trainingcited by 68
  29. 2021Look at the Variance! Efficient Black-box Explanations with Sobol-based Sensitivity Analysiscited by 65
  30. 2019Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learningcited by 60
  31. 2021Learning Task Informed Abstractionscited by 58
  32. 2019Obstacle Tower: A Generalization Challenge in Vision, Control, and Planningcited by 54
  33. 2022Look Closer: Bridging Egocentric and Third-Person Views with Transformers for Robotic Manipulationcited by 49
  34. 2021SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policiescited by 42
  35. 2018Natural Environment Benchmarks for Reinforcement Learningcited by 32
  36. 2021DRIBO: Robust Deep Reinforcement Learning via Multi-View Information Bottleneckcited by 31
  37. 2021Unsupervised Visual Attention and Invariance for Reinforcement Learningcited by 30
  38. 2022Don't Touch What Matters: Task-Aware Lipschitz Data Augmentation for Visual Reinforcement Learningcited by 21

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 8; 2024: 7; 2025: 11; 2026: 5.

32 papers.

Sort:

By area:

  1. 2025Latent Action Learning Requires Supervision in the Presence of Distractorscited by 40
  2. 2024Learning to Manipulate Anywhere: A Visual Generalizable Framework For Reinforcement Learningcited by 36
  3. 2023RL-ViGen: A Reinforcement Learning Benchmark for Visual Generalizationcited by 27
  4. 2023Refining Diffusion Planner for Reliable Behavior Synthesis by Automatic Detection of Infeasible Planscited by 27
  5. 2023From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscapecited by 18
  6. 2025A Survey of State Representation Learning for Deep Reinforcement Learningcited by 17
  7. 2023MoVie: Visual Model-Based Policy Adaptation for View Generalizationcited by 15
  8. 2023Normalization Enhances Generalization in Visual Reinforcement Learningcited by 13
  9. 2023Generalizable Visual Reinforcement Learning with Segment Anything Modelcited by 12
  10. 2024ViSaRL: Visual Reinforcement Learning Guided by Human Saliencycited by 12
  11. 2023MaDi: Learning to Mask Distractions for Generalization in Visual Deep Reinforcement Learningcited by 9
  12. 2024Understanding What Affects the Generalization Gap in Visual Reinforcement Learning: Theory and Empirical Evidencecited by 5
20 more
  1. 2024Focus On What Matters: Separated Models For Visual-Based RL Generalizationcited by 5
  2. 2025HASARD: A Benchmark for Vision-Based Safe Reinforcement Learning in Embodied Agentscited by 4
  3. 2022A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challengescited by 3
  4. 2023Learning Generalizable Agents via Saliency-Guided Features Decorrelationcited by 3
  5. 2024The Overfocusing Bias of Convolutional Neural Networks: A Saliency-Guided Regularization Approachcited by 2
  6. 2025Deep Reinforcement Learning via Object-Centric Attentioncited by 2
  7. 2024SeMOPO: Learning High-quality Model and Policy from Low-quality Offline Visual Datasetscited by 1
  8. 2025Salience-Invariant Consistent Policy Learning for Generalization in Visual Reinforcement Learningcited by 1
  9. 2024BET: Explaining Deep Reinforcement Learning through The Error-Prone Decisionscited by 0
  10. 2025Self-Consistent Model-based Adaptation for Visual Reinforcement Learningcited by 0
  11. 2025Common Benchmarks Undervalue the Generalization Power of Programmatic Policiescited by 0
  12. 2025Multimodal Fusion for Sim2real Transfer in Visual Reinforcement Learningcited by 0
  13. 2025SegDAC: Visual Generalization in Reinforcement Learning via Dynamic Object Tokenscited by 0
  14. 2025DeGuV: Depth-Guided Visual Reinforcement Learning for Generalization and Interpretability in Manipulationcited by 0
  15. 2025Gaze on the Prize: Shaping Visual Attention with Return-Guided Contrastive Learningcited by 0
  16. 2026KAGE-Bench: Fast Known-Axis Visual Generalization Evaluation for Reinforcement Learningcited by 0
  17. 2026Sparse Masked Attention Policies for Reliable Generalizationcited by 0
  18. 2026Boosting deep Reinforcement Learning using pretraining with Logical Optionscited by 0
  19. 2026What to Ignore, What to React: Visually Robust RL Fine-Tuning of VLA Modelscited by 0
  20. 2026Task-Relevant Representation Decoupling for Visual Reinforcement Learning Generalizationcited by 0