mlsanityData to 14 Oct 2026

UNIQ: Conformal Calibration for Adaptive Conservatism in Offline Reinforcement Learning

arXiv 2606.07592 ↗First posted 28 May 2026cs.LGcited by 1

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning requires careful conservatism to mitigate distribution shift, yet most existing methods apply a fixed penalty uniformly across all states regardless of local data coverage. We present UNIQ (Uncertainty-Informed Quantile), an offline RL method that introduces state-adaptive conservatism through conformally calibrated uncertainty estimation. Built on the Implicit Q-Learning (IQL) backbone, UNIQ trains a multi-expectile value ensemble, computes distribution-free uncertainty estimates using split conformal prediction, and maps the resulting signal to a state-dependent expectile that relaxes conservatism in well-covered regions while strengthening it in uncertain regions near the data frontier. On D4RL MuJoCo benchmarks, UNIQ consistently improves over IQL, with the largest gains observed on Walker2d and replay-heavy tasks. At the same time, UNIQ operates at near-IQL memory cost (approximately 250 MB peak VRAM), providing roughly a 10x reduction compared to EDAC. Rather than pursuing overall state-of-the-art performance, we position UNIQ as a practical mechanism contribution that improves the performance-efficiency trade-off in offline reinforcement learning.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 24 of its 27 references that are in the map. First posted on arXiv 28 May 2026.

Builds on

24 of 27 references.

Closest ancestors · 16

Same direction, newest first

  1. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library13 Oct
  2. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  3. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  4. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  5. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  6. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  7. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  8. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  9. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  10. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  11. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  12. 2020MOPO: Model-based Offline Policy Optimization27 May
4 more
  1. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  4. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 8

  1. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  3. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  5. 2016Distribution-Free Predictive Inference For Regressioncited by 668
  6. 2019Conformalized Quantile Regressioncited by 629
  7. 2019Conformal Prediction Under Covariate Shiftcited by 519
  8. 2021Adaptive Conformal Inference Under Distribution Shiftcited by 375

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Convex-Hull-Neighborhood Smooth Dual Generalization: Controlling Local Correction Propagation in Offline RLcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Conformal Prediction Beyond the Horizon: Distribution-Free Inference for Policy Evaluation
  2. 2026Epistemic Robust Offline Reinforcement Learning
  3. 2024Echoes of Socratic Doubt: Embracing Uncertainty in Calibrated Evidential Reinforcement Learning
  4. 2023A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learning
  5. 2026Bayesian Conservative Policy Optimization (BCPO): A Novel Uncertainty-Calibrated Offline Reinforcement Learning with Credible Lower Bounds
  6. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruption
  7. 2022Incorporating Explicit Uncertainty Estimates into Deep Offline Reinforcement Learning
  8. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning