Data to 7 Oct 2026

Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning

arXiv 2101.05265 ↗First posted 13 Jan 2021cs.LG cs.AI stat.MLcited by 123

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning methods trained on few environments rarely learn policies that generalize to unseen environments. To improve generalization, we incorporate the inherent sequential structure in reinforcement learning into the representation learning process. This approach is orthogonal to recent approaches, which rarely exploit this structure explicitly. Specifically, we introduce a theoretically motivated policy similarity metric (PSM) for measuring behavioral similarity between states. PSM assigns high similarity to states for which the optimal policies in those states as well as in future states are similar. We also present a contrastive representation learning procedure to embed any state similarity metric, which we instantiate with PSM to obtain policy similarity embeddings (PSEs). We demonstrate that PSEs improve generalization on diverse benchmarks, including LQR with spurious correlations, a jumping task from pixels, and Distracting DM Control Suite.

Key facts. Cited by 123 papers in the map (data through 7 Oct 2026). Builds on 48 of its 67 references that are in the map. First posted on arXiv 13 Jan 2021.

Builds on

48 of 67 references.

References · 48

2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
2018Representation Learning with Contrastive Predictive Codingcited by 7,662
2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
2019Invariant Risk Minimizationcited by 1,832
2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
2017Learning to Generalize: Meta-Learning for Domain Generalizationcited by 762
2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
2020Reinforcement Learning with Augmented Datacited by 452

36 more

2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
2019Implicit Regularization in Deep Matrix Factorizationcited by 402
2020dm_control: Software and Tasks for Continuous Controlcited by 368
2018Quantifying Generalization in Reinforcement Learningcited by 345
2017Implicit Regularization in Matrix Factorizationcited by 318
2020Decoupling Representation Learning from Reinforcement Learningcited by 263
2020How Neural Networks Extrapolate: From Feedforward to Graph Neural Networkscited by 218
2018A Study on Overfitting in Deep Reinforcement Learningcited by 170
2019Scalable methods for computing state similarity in deterministic Markov Decision Processescited by 133
2018Assessing Generalization in Deep Reinforcement Learningcited by 132
2018A Tour of Reinforcement Learning: The View from Continuous Controlcited by 129
2018Generalization and Regularization in DQNcited by 117
2020Invariant Causal Prediction for Block MDPscited by 104
2019Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learningcited by 102
2019Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneckcited by 101
2017Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learningcited by 97
2021The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixelscited by 97
2019Observational Overfitting in Reinforcement Learningcited by 90
2018A Dissection of Overfitting and Generalization in Continuous Reinforcement Learningcited by 86
2018Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generationcited by 80
2020Deep Reinforcement and InfoMax Learningcited by 79
2020Automatic Data Augmentation for Generalization in Deep Reinforcement Learningcited by 71
2017Towards Generalization and Simplicity in Continuous Controlcited by 65
2020Plannable Approximations to MDP Homomorphisms: Equivariance under Actionscited by 61
2020Predictive Information Accelerates Learning in RLcited by 56
2019Obstacle Tower: A Generalization Challenge in Vision, Control, and Planningcited by 54
2020Neuroevolution of Self-Interpretable Agentscited by 48
2019Learning to Generalize from Sparse and Underspecified Rewardscited by 39
2020Invariant Policy Optimization: Towards Stronger Generalization in Reinforcement Learningcited by 37
2018Natural Environment Benchmarks for Reinforcement Learningcited by 32
2017Robust and Efficient Transfer Learning with Hidden-Parameter Markov Decision Processescited by 28
2020Rotation, Translation, and Cropping for Zero-Shot Generalizationcited by 23
2020Enforcing Predictive Invariance across Structured Biomedical Domainscited by 23
2018Measuring and Characterizing Generalization in Deep Reinforcement Learningcited by 20
2020Generalized Hidden Parameter MDPs Transferable Model-based RL in a Handful of Trialscited by 20
2018Learning Invariances for Policy Generalizationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 25; 2022: 24; 2023: 33; 2024: 15; 2025: 14; 2026: 12.

123 papers.

Sort:

By area:

2022Evolving Curricula with Regret-Based Environment Designcited by 166
2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 145
2021Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observabilitycited by 102
2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 101
2021Representation Matters: Offline Pretraining for Sequential Decision Makingcited by 90
2021Decoupling Value and Policy for Generalization in Reinforcement Learningcited by 81
2021DreamerPro: Reconstruction-Free Model-Based Reinforcement Learning with Prototypical Representationscited by 73
2021Learning Task Informed Abstractionscited by 58
2022Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learningcited by 58
2021Towards Robust Bisimulation Metric Learningcited by 49

38 more

2021AdaRL: What, Where, and How to Adapt in Transfer Reinforcement Learningcited by 46
2021MICo: Improved representations via sampling-based state similarity for Markov decision processescited by 43
2023Proto-Value Networks: Scaling Representation Learning with Auxiliary Taskscited by 41
2022What Makes Good Contrastive Learning on Small-Scale Wearable-based Tasks?cited by 39
2022Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learningcited by 38
2021Provable Representation Learning for Imitation with Contrastive Fourier Featurescited by 34
2021Proper Value Equivalencecited by 33
2024Run-time Observation Interventions Make Vision-Language-Action Models More Visually Robustcited by 33
2022Look where you look! Saliency-guided Q-networks for generalization in visual Reinforcement Learningcited by 32
2021DRIBO: Robust Deep Reinforcement Learning via Multi-View Information Bottleneckcited by 31
2023On the Importance of Exploration for Generalization in Reinforcement Learningcited by 30
2022On the Generalization of Representations in Reinforcement Learningcited by 28
2023Conditional Mutual Information for Disentangled Representations in Reinforcement Learningcited by 27
2021Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoningcited by 25
2023The Generalization Gap in Offline Reinforcement Learningcited by 25
2021Learning Representations for Pixel-based Control: What Matters and Why?cited by 24
2022Investigating the Properties of Neural Network Representations in Reinforcement Learningcited by 24
2022Does Self-supervised Learning Really Improve Reinforcement Learning from Pixels?cited by 24
2022Continuous MDP Homomorphisms and Homomorphic Policy Gradientcited by 24
2021Towards Better Laplacian Representation in Reinforcement Learning with Generalized Graph Drawingcited by 22
2021Cross-Trajectory Representation Learning for Zero-Shot Generalization in RLcited by 20
2021CoBERL: Contrastive BERT for Reinforcement Learningcited by 20
2021Towards Deeper Deep Reinforcement Learning with Spectral Normalizationcited by 19
2022Temporal Disentanglement of Representations for Improved Generalisation in Reinforcement Learningcited by 19
2022Pretraining in Deep Reinforcement Learning: A Surveycited by 18
2023Look Beneath the Surface: Exploiting Fundamental Symmetry for Sample-Efficient Offline RLcited by 18
2021Improving Zero-shot Generalization in Offline Reinforcement Learning using Generalized Similarity Functionscited by 17
2022Open Source Vizier: Distributed Infrastructure and API for Reliable and Flexible Blackbox Optimizationcited by 17
2025A Survey of State Representation Learning for Deep Reinforcement Learningcited by 17
2022Learning Task-relevant Representations for Generalization via Characteristic Functions of Reward Sequence Distributionscited by 15
2022Contrastive Value Learning: Implicit Models for Simple Offline RLcited by 15
2023Robust Representation Learning by Clustering with Bisimulation Metrics for Visual Reinforcement Learning with Distractionscited by 13
2023Why LLMs Hallucinate, and How to Get (Evidential) Closure: Perceptual, Intensional, and Extensional Learning for Faithful Natural Language Generationcited by 13
2021Learning Domain Invariant Representations in Goal-conditioned Block MDPscited by 12
2021Learning Temporally-Consistent Representations for Data-Efficient Reinforcement Learningcited by 10
2022Reachability-Aware Laplacian Representation in Reinforcement Learningcited by 10
2023Understanding and Addressing the Pitfalls of Bisimulation-based Representations in Offline Reinforcement Learningcited by 10
2023Generalization through Diversity: Improving Unsupervised Environment Designcited by 9

50 most cited of 123 All 123 (JSON)