mlsanityData to 7 Oct 2026

Improving Zero-shot Generalization in Offline Reinforcement Learning using Generalized Similarity Functions

arXiv 2111.14629 ↗First posted 29 Nov 2021cs.LG cs.AIcited by 17

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Reinforcement learning (RL) agents are widely used for solving complex sequential decision making tasks, but still exhibit difficulty in generalizing to scenarios not seen during training. While prior online approaches demonstrated that using additional signals beyond the reward function can lead to better generalization capabilities in RL agents, i.e. using self-supervised learning (SSL), they struggle in the offline RL setting, i.e. learning from a static dataset. We show that performance of online algorithms for generalization in RL can be hindered in the offline setting due to poor estimation of similarity between observations. We propose a new theoretically-motivated framework called Generalized Similarity Functions (GSF), which uses contrastive learning to train an offline RL agent to aggregate observations based on the similarity of their expected future behavior, where we quantify this similarity using generalized value functions. We show that GSF is general enough to recover existing SSL objectives while also improving zero-shot generalization performance on a complex offline RL benchmark, offline Procgen.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 58 of its 90 references that are in the map. First posted on arXiv 29 Nov 2021.

Builds on

58 of 90 references.

Closest ancestors · 9

Same direction, newest first

  1. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  2. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning10 Mar
  3. 2021Instabilities of Offline RL with Pre-Trained Neural Representation8 Mar
  4. 2020Provably Good Batch Reinforcement Learning Without Great Exploration16 Jul
  5. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning24 Jun
  6. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  7. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  9. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 49

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  6. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  7. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  8. 2020Supervised Contrastive Learningcited by 2,803
  9. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  10. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  11. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  12. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
37 more
  1. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  2. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  3. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  4. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  5. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  6. 2018Quantifying Generalization in Reinforcement Learningcited by 345
  7. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
  8. 2019Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examplescited by 312
  9. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  10. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  11. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  12. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 204
  13. 2017DARLA: Improving Zero-Shot Transfer in Reinforcement Learningcited by 147
  14. 2018Multi-task Deep Reinforcement Learning with PopArtcited by 138
  15. 2019Scalable methods for computing state similarity in deterministic Markov Decision Processescited by 133
  16. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
  17. 2017Value Prediction Networkcited by 117
  18. 2021Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videoscited by 116
  19. 2019Kinematic State Abstraction and Provably Efficient Rich-Observation Reinforcement Learningcited by 114
  20. 2018Count-Based Exploration with the Successor Representationcited by 112
  21. 2021Learning One Representation to Optimize All Rewardscited by 110
  22. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
  23. 2021The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixelscited by 97
  24. 2020PC-PG: Policy Cover Directed Exploration for Provable Policy Gradient Learningcited by 91
  25. 2019Observational Overfitting in Reinforcement Learningcited by 90
  26. 2021Decoupling Value and Policy for Generalization in Reinforcement Learningcited by 81
  27. 2020Deep Reinforcement and InfoMax Learningcited by 79
  28. 2020A Universal Representation Transformer Layer for Few-Shot Image Classificationcited by 65
  29. 2021Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environmentcited by 41
  30. 2021Heuristic-Guided Reinforcement Learningcited by 41
  31. 2021An Exponential Lower Bound for Linearly-Realizable MDPs with Constant Suboptimality Gapcited by 40
  32. 2018Truncated Horizon Policy Search: Combining Reinforcement Learning & Imitation Learningcited by 39
  33. 2020Generalization bounds for deep learningcited by 38
  34. 2021Provable Representation Learning for Imitation with Contrastive Fourier Featurescited by 34
  35. 2020Multi-label Contrastive Predictive Codingcited by 32
  36. 2021Cross-Trajectory Representation Learning for Zero-Shot Generalization in RLcited by 20
  37. 2018Learning Invariances for Policy Generalizationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 4; 2024: 6; 2025: 5; 2026: 1.

17 papers.

Sort:

By area:

  1. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 82
  2. 2023What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?cited by 36
  3. 2023On the Importance of Exploration for Generalization in Reinforcement Learningcited by 30
  4. 2022Semi-Supervised Offline Reinforcement Learning with Action-Free Trajectoriescited by 27
  5. 2023The Generalization Gap in Offline Reinforcement Learningcited by 25
  6. 2024V2X-DGW: Domain Generalization for Multi-agent Perception under Adverse Weather Conditionscited by 12
  7. 2023On the Opportunities and Challenges of Offline Reinforcement Learning for Recommender Systemscited by 10
  8. 2025Provable Zero-Shot Generalization in Offline Reinforcement Learningcited by 3
  9. 2025Filtering Learning Histories Enhances In-Context Reinforcement Learningcited by 3
  10. 2024Ensemble Successor Representations for Task Generalization in Offline-to-Online Reinforcement Learningcited by 1
  11. 2024Random Policy Enables In-Context Reinforcement Learning within Trust Horizonscited by 1
  12. 2024The Synergy of LLMs & RL Unlocks Offline Learning of Generalizable Language-Conditioned Policies with Low-fidelity Datacited by 1
5 more
  1. 2024Accelerating Task Generalisation with Multi-Level Skill Hierarchiescited by 0
  2. 2025Zero-Shot Action Generalization with Limited Observationscited by 0
  3. 2025LLM-Driven Policy Diffusion: Enhancing Generalization in Offline Reinforcement Learningcited by 0
  4. 2025Forecasting in Offline Reinforcement Learning for Non-stationary Environmentscited by 0
  5. 2026Generalization in offline RL: The structure is more important than the amount of pessimismcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Doubly Mild Generalization for Offline Reinforcement Learning
  2. 2026Improving Zero-Shot Offline RL via Behavioral Task Sampling
  3. 2022Domain Generalization for Robust Model-Based Offline Reinforcement Learning
  4. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
  5. 2026Paying Less Generalization Tax: A Cross-Domain Generalization Study of RL Training for LLM Agents
  6. 2026Out-of-Distribution Generalisation with Sequence Models in Offline Multi-Agent Reinforcement Learning
  7. 2025Synthetic Data is Sufficient for Zero-Shot Visual Generalization from Offline Data
  8. 2024Foundation Policies with Hilbert Representations