# Look where you look! Saliency-guided Q-networks for generalization in visual Reinforcement Learning

arXiv: https://arxiv.org/abs/2209.09203 · HTML: https://mlsanity.com/paper/2209.09203/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 56 of its 71 references that are in the map.
- First posted on arXiv 16 Sep 2022.

## Builds on (most cited first, 30 of 56)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks](https://mlsanity.com/paper/1710.11063/) (2017)
- [RISE: Randomized Input Sampling for Explanation of Black-box Models](https://mlsanity.com/paper/1806.07421/) (2018)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels](https://mlsanity.com/paper/2004.13649/) (2020)
- [Domain Generalization with MixStyle](https://mlsanity.com/paper/2104.02008/) (2021)
- [Leveraging Procedural Generation to Benchmark Reinforcement Learning](https://mlsanity.com/paper/1912.01588/) (2019)
- [Reinforcement Learning with Augmented Data](https://mlsanity.com/paper/2004.14990/) (2020)
- [Learning Invariant Representations for Reinforcement Learning without Reconstruction](https://mlsanity.com/paper/2006.10742/) (2020)
- [Quantifying Generalization in Reinforcement Learning](https://mlsanity.com/paper/1812.02341/) (2018)
- [A Theory of Regularized Markov Decision Processes](https://mlsanity.com/paper/1901.11275/) (2019)
- [Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents](https://mlsanity.com/paper/1709.06009/) (2017)
- [Prioritized Level Replay](https://mlsanity.com/paper/2010.03934/) (2020)
- [A Study on Overfitting in Deep Reinforcement Learning](https://mlsanity.com/paper/1804.06893/) (2018)
- [DARLA: Improving Zero-Shot Transfer in Reinforcement Learning](https://mlsanity.com/paper/1707.08475/) (2017)
- [Visualizing and Understanding Atari Agents](https://mlsanity.com/paper/1711.00138/) (2017)
- [Generalization in Reinforcement Learning by Soft Data Augmentation](https://mlsanity.com/paper/2011.13389/) (2020)
- [Assessing Generalization in Deep Reinforcement Learning](https://mlsanity.com/paper/1810.12282/) (2018)
- [Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning](https://mlsanity.com/paper/2101.05265/) (2021)

## Built on by (most cited first, 30 of 34)

- [Latent Action Learning Requires Supervision in the Presence of Distractors](https://mlsanity.com/paper/2502.00379/) (2025)
- [Learning to Manipulate Anywhere: A Visual Generalizable Framework For Reinforcement Learning](https://mlsanity.com/paper/2407.15815/) (2024)
- [RL-ViGen: A Reinforcement Learning Benchmark for Visual Generalization](https://mlsanity.com/paper/2307.10224/) (2023)
- [Refining Diffusion Planner for Reliable Behavior Synthesis by Automatic Detection of Infeasible Plans](https://mlsanity.com/paper/2310.19427/) (2023)
- [From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscape](https://mlsanity.com/paper/2312.10868/) (2023)
- [A Survey of State Representation Learning for Deep Reinforcement Learning](https://mlsanity.com/paper/2506.17518/) (2025)
- [MoVie: Visual Model-Based Policy Adaptation for View Generalization](https://mlsanity.com/paper/2307.00972/) (2023)
- [Normalization Enhances Generalization in Visual Reinforcement Learning](https://mlsanity.com/paper/2306.00656/) (2023)
- [Generalizable Visual Reinforcement Learning with Segment Anything Model](https://mlsanity.com/paper/2312.17116/) (2023)
- [ViSaRL: Visual Reinforcement Learning Guided by Human Saliency](https://mlsanity.com/paper/2403.10940/) (2024)
- [MaDi: Learning to Mask Distractions for Generalization in Visual Deep Reinforcement Learning](https://mlsanity.com/paper/2312.15339/) (2023)
- [Focus On What Matters: Separated Models For Visual-Based RL Generalization](https://mlsanity.com/paper/2410.10834/) (2024)
- [HASARD: A Benchmark for Vision-Based Safe Reinforcement Learning in Embodied Agents](https://mlsanity.com/paper/2503.08241/) (2025)
- [Understanding What Affects the Generalization Gap in Visual Reinforcement Learning: Theory and Empirical Evidence](https://mlsanity.com/paper/2402.02701/) (2024)
- [Learning Generalizable Agents via Saliency-Guided Features Decorrelation](https://mlsanity.com/paper/2310.05086/) (2023)
- [A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challenges](https://mlsanity.com/paper/2211.06665/) (2022)
- [The Overfocusing Bias of Convolutional Neural Networks: A Saliency-Guided Regularization Approach](https://mlsanity.com/paper/2409.17370/) (2024)
- [Deep Reinforcement Learning via Object-Centric Attention](https://mlsanity.com/paper/2504.03024/) (2025)
- [SeMOPO: Learning High-quality Model and Policy from Low-quality Offline Visual Datasets](https://mlsanity.com/paper/2406.09486/) (2024)
- [Salience-Invariant Consistent Policy Learning for Generalization in Visual Reinforcement Learning](https://mlsanity.com/paper/2502.08336/) (2025)
- [Self-Consistent Model-based Adaptation for Visual Reinforcement Learning](https://mlsanity.com/paper/2502.09923/) (2025)
- [Gaze on the Prize: Shaping Visual Attention with Return-Guided Contrastive Learning](https://mlsanity.com/paper/2510.08442/) (2025)
- [What to Ignore, What to React: Visually Robust RL Fine-Tuning of VLA Models](https://mlsanity.com/paper/2605.13105/) (2026)
- [BET: Explaining Deep Reinforcement Learning through The Error-Prone Decisions](https://mlsanity.com/paper/2401.07263/) (2024)
- [A Visual Reinforcement Learning-Based Separate Primitive Policy for Peg-in-Hole Tasks](https://mlsanity.com/paper/2504.14820/) (2025)
- [Common Benchmarks Undervalue the Generalization Power of Programmatic Policies](https://mlsanity.com/paper/2506.14162/) (2025)
- [Multimodal Fusion for Sim2real Transfer in Visual Reinforcement Learning](https://mlsanity.com/paper/2507.09180/) (2025)
- [SegDAC: Visual Generalization in Reinforcement Learning via Dynamic Object Tokens](https://mlsanity.com/paper/2508.09325/) (2025)
- [DeGuV: Depth-Guided Visual Reinforcement Learning for Generalization and Interpretability in Manipulation](https://mlsanity.com/paper/2509.04970/) (2025)
- [KAGE-Bench: Fast Known-Axis Visual Generalization Evaluation for Reinforcement Learning](https://mlsanity.com/paper/2601.14232/) (2026)

## Abstract (arXiv)

Deep reinforcement learning policies, despite their outstanding efficiency in simulated visual control tasks, have shown disappointing ability to generalize across disturbances in the input training images. Changes in image statistics or distracting background elements are pitfalls that prevent generalization and real-world applicability of such control policies. We elaborate on the intuition that a good visual policy should be able to identify which pixels are important for its decision, and preserve this identification of important sources of information across images. This implies that training of a policy with small generalization gap should focus on such important pixels and ignore the others. This leads to the introduction of saliency-guided Q-networks (SGQN), a generic method for visual reinforcement learning, that is compatible with any value function learning method. SGQN vastly improves the generalization capability of Soft Actor-Critic agents and outperforms existing stateof-the-art methods on the Deepmind Control Generalization benchmark, setting a new reference in terms of training efficiency, generalization gap, and policy interpretability.