Learning Generalizable Agents via Saliency-Guided Features Decorrelation
Not on the map. Why?
Abstract (arXiv)
In visual-based Reinforcement Learning (RL), agents often struggle to generalize well to environmental variations in the state space that were not observed during training. The variations can arise in both task-irrelevant features, such as background noise, and task-relevant features, such as robot configurations, that are related to the optimal decisions. To achieve generalization in both situations, agents are required to accurately understand the impact of changed features on the decisions, i.e., establishing the true associations between changed features and decisions in the policy model. However, due to the inherent correlations among features in the state space, the associations between features and decisions become entangled, making it difficult for the policy to distinguish them. To this end, we propose Saliency-Guided Features Decorrelation (SGFD) to eliminate these correlations through sample reweighting. Concretely, SGFD consists of two core techniques: Random Fourier Functions (RFF) and the saliency map. RFF is utilized to estimate the complex non-linear correlations in high-dimensional images, while the saliency map is designed to identify the changed features. Under the guidance of the saliency map, SGFD employs sample reweighting to minimize the estimated correlations related to changed features, thereby achieving decorrelation in visual RL tasks. Our experimental results demonstrate that SGFD can generalize well on a wide range of test environments and significantly outperforms state-of-the-art methods in handling both task-irrelevant variations and task-relevant variations.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 43 of its 49 references that are in the map. First posted on arXiv 8 Oct 2023.
Builds on
43 of 49 references.
References · 43
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2017The Kinetics Human Action Video Datasetcited by 1,894
- 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
- 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
- 2018Sanity Checks for Saliency Mapscited by 1,149
- 2018DeepMind Control Suitecited by 935
- 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
- 2017One pixel attack for fooling deep neural networkscited by 579
- 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
- 2020Reinforcement Learning with Augmented Datacited by 452
- 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
31 more
- 2023A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPTcited by 364
- 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
- 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
- 2018Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experiencecited by 203
- 2021Deep Stable Learning for Out-Of-Distribution Generalizationcited by 153
- 2020Generalization in Reinforcement Learning by Soft Data Augmentationcited by 132
- 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
- 2020CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learningcited by 109
- 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
- 2020Invariant Causal Prediction for Block MDPscited by 104
- 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 101
- 2020Deep Reinforcement and InfoMax Learningcited by 79
- 2017Approximate Kernel-based Conditional Independence Tests for Fast Non-Parametric Causal Discoverycited by 76
- 2019Stable Learning via Sample Reweightingcited by 70
- 2020Stable Prediction with Model Misspecification and Agnostic Distribution Shiftcited by 62
- 2019Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learningcited by 60
- 2022Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learningcited by 58
- 2018Stable Prediction across Unknown Environmentscited by 55
- 2023SMART: Self-supervised Multi-task pretrAining with contRol Transformerscited by 44
- 2022Look where you look! Saliency-guided Q-networks for generalization in visual Reinforcement Learningcited by 32
- 2020S3K: Self-Supervised Semantic Keypoints for Robotic Manipulation via Multi-View Consistencycited by 28
- 2022Don't Touch What Matters: Task-Aware Lipschitz Data Augmentation for Visual Reinforcement Learningcited by 21
- 2021Cross-Trajectory Representation Learning for Zero-Shot Generalization in RLcited by 20
- 2021Domain Adversarial Reinforcement Learningcited by 19
- 2022Temporal Disentanglement of Representations for Improved Generalisation in Reinforcement Learningcited by 19
- 2019Affordance Learning for End-to-End Visuomotor Robot Controlcited by 18
- 2021A Theoretical Analysis on Independence-driven Importance Weighting for Covariate-shift Generalizationcited by 18
- 2022Stable Learning via Sparse Variable Independencecited by 13
- 2022Local Feature Swapping for Generalization in Reinforcement Learningcited by 9
- 2022Distributional Reward Estimation for Effective Multi-Agent Deep Reinforcement Learningcited by 6
- 2022Learning Generalizable Representations for Reinforcement Learning via Adaptive Meta-learner of Behavioral Similaritiescited by 4
Built on by
Citing papers per year: 2024: 1; 2025: 2.
3 papers.