mlsanityData to 7 Oct 2026

Epistemic Robust Offline Reinforcement Learning

arXiv 2604.07072 ↗First posted 8 Apr 2026cs.LGcited by 1

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning learns policies from fixed datasets without further environment interaction. A key challenge in this setting is epistemic uncertainty, arising from limited or biased data coverage, particularly when the behavior policy systematically avoids certain actions. This can lead to inaccurate value estimates and unreliable generalization. Ensemble-based methods like SAC-N mitigate this by conservatively estimating Q-values using the ensemble minimum, but they require large ensembles and often conflate epistemic with aleatoric uncertainty. To address these limitations, we propose a unified and generalizable framework that replaces discrete ensembles with compact uncertainty sets over Q-values. %We further introduce an Epinet based model that directly shapes the uncertainty sets to optimize the cumulative reward under the robust Bellman objective without relying on ensembles. We also introduce a benchmark for evaluating offline RL algorithms under risk-sensitive behavior policies, and demonstrate that our method achieves improved robustness and generalization over ensemble-based baselines across both tabular and continuous state domains.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 32 of its 46 references that are in the map. First posted on arXiv 8 Apr 2026.

Builds on

32 of 46 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024Doubly Mild Generalization for Offline Reinforcement Learning12 Nov
  2. 2024Offline Reinforcement Learning with OOD State Correction and OOD Action Suppression25 Oct
  3. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexity11 Aug
  4. 2022Offline RL Policies Should be Trained to be Adaptive5 Jul
  5. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?12 Apr
  6. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  7. 2021A Dataset Perspective on Offline Reinforcement Learning8 Nov
  8. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  9. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  10. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  11. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  12. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
3 more
  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  2. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  3. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 17

  1. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  2. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  5. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  6. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  7. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 368
  8. 2019Deep Evidential Regressioncited by 354
  9. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  10. 2019Soft Actor-Critic for Discrete Action Settingscited by 161
  11. 2021Epistemic Neural Networkscited by 120
  12. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 78
5 more
  1. 2024Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learningcited by 29
  2. 2020Distributionally robust stochastic programs with side information based on trimmings -- Extended versioncited by 19
  3. 2023Predict-then-Calibrate: A New Perspective of Robust Contextual LPcited by 17
  4. 2019Dynamic optimization with side informationcited by 14
  5. 2021Robustifying Conditional Portfolio Decisions via Optimal Transportcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026When Policies Cannot Be Retrained: A Unified Closed-Form View of Post-Training Steering in Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Balancing policy constraint and ensemble size in uncertainty-based offline reinforcement learning
  2. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters
  3. 2024Diverse Randomized Value Functions: A Provably Pessimistic Approach for Offline Reinforcement Learning
  4. 2024Pessimistic Value Iteration for Multi-Task Data Sharing in Offline Reinforcement Learning
  5. 2026Bayesian Conservative Policy Optimization (BCPO): A Novel Uncertainty-Calibrated Offline Reinforcement Learning with Credible Lower Bounds
  6. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning
  7. 2025Evaluation-Time Policy Switching for Offline Reinforcement Learning
  8. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning