mlsanityData to 14 Oct 2026

Generalization in offline RL: The structure is more important than the amount of pessimism

arXiv 2607.02288 ↗First posted 2 Jul 2026cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

While pessimism counteracts overestimation bias in offline reinforcement learning (RL), being overly conservative has been associated with hindering certain forms of generalization. However, in this paper we demonstrate that being overly pessimistic does not inherently prevent optimal generalization in contextual MDPs (CMDPs). Instead, we argue successful generalization depends not on the amount of pessimism, but whether the pessimistic structure respects the underlying symmetries of the optimal solution. We prove that a mildly pessimistic, non-symmetric value function can generalize worse than an overly pessimistic, symmetric one. In offline RL, the structure of the pessimism is determined by the structure of the dataset coverage. As such, enforcing a symmetric value function can be non-trivial, and might require techniques such as data augmentation (DA). Inspired by our theoretical results, we argue that DA can best be applied through a consistency loss during policy extraction, rather than the common practice of (regular) offline training on an augmented dataset. This is empirically validated using IQL and CQL on a rotationally symmetric reacher environment.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 34 of its 44 references that are in the map. First posted on arXiv 2 Jul 2026.

Builds on

34 of 44 references.

Closest ancestors · 19

Same direction, newest first

  1. 2025Provable Zero-Shot Generalization in Offline Reinforcement Learning11 Mar
  2. 2024Goal-Conditioned Data Augmentation for Offline Reinforcement Learning29 Dec
  3. 2024Doubly Mild Generalization for Offline Reinforcement Learning12 Nov
  4. 2024Is Value Learning Really the Main Bottleneck in Offline RL?13 Jun
  5. 2024Strategically Conservative Q-Learning6 Jun
  6. 2024GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learning27 May
  7. 2023The Generalization Gap in Offline Reinforcement Learning10 Dec
  8. 2023Guided Data Augmentation for Offline Reinforcement Learning and Imitation Learning27 Oct
  9. 2023Equivariant Data Augmentation for Generalization in Offline Reinforcement Learning14 Sep
  10. 2023Look Beneath the Surface: Exploiting Fundamental Symmetry for Sample-Efficient Offline RL7 Jun
  11. 2022S2P: State-conditioned Image Synthesis for Data Augmentation in Offline Reinforcement Learning30 Sep
  12. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
7 more
  1. 2021Improving Zero-shot Generalization in Offline Reinforcement Learning using Generalized Similarity Functions29 Nov
  2. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  3. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  4. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  5. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning10 Mar
  6. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 15

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  3. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  4. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  5. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  6. 2014Learning with Pseudo-Ensemblescited by 241
  7. 2015Contextual Markov Decision Processescited by 220
  8. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 170
  9. 2020Generalization in Reinforcement Learning by Soft Data Augmentationcited by 137
  10. 2023What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?cited by 36
  11. 2022Sample Efficiency of Data Augmentation Consistency Regularizationcited by 28
  12. 2023Fantastic Generalization Measures are Nowhere to be Foundcited by 19
3 more
  1. 2024Emergent Equivariance in Deep Ensemblescited by 18
  2. 2024A Recipe for Unbounded Data Augmentation in Visual Reinforcement Learningcited by 14
  3. 2025How Ensembles of Distilled Policies Improve Generalisation in Reinforcement Learningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2020Is Pessimism Provably Efficient for Offline RL?
  2. 2022State-Aware Proximal Pessimistic Algorithms for Offline Reinforcement Learning
  3. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning
  4. 2021Optimality Inductive Biases and Agnostic Guidelines for Offline Reinforcement Learning
  5. 2025Semi-pessimistic Reinforcement Learning
  6. 2022Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processes
  7. 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief
  8. 2026Distorted Distributional Policy Evaluation for Offline Reinforcement Learning