mlsanityData to 14 Oct 2026

Hallucinated Adversarial Control for Conservative Offline Policy Evaluation

arXiv 2303.01076 ↗First posted 2 Mar 2023cs.LG cs.AI stat.MLcited by 13

Not on the map. Why?

Abstract (arXiv)

We study the problem of conservative off-policy evaluation (COPE) where given an offline dataset of environment interactions, collected by other agents, we seek to obtain a (tight) lower bound on a policy's performance. This is crucial when deciding whether a given policy satisfies certain minimal performance/safety criteria before it can be deployed in the real world. To this end, we introduce HAMBO, which builds on an uncertainty-aware learned model of the transition dynamics. To form a conservative estimate of the policy's performance, HAMBO hallucinates worst-case trajectories that the policy may take, within the margin of the models' epistemic confidence regions. We prove that the resulting COPE estimates are valid lower bounds, and, under regularity conditions, show their convergence to the true expected return. Finally, we discuss scalable variants of our approach based on Bayesian Neural Networks and empirically demonstrate that they yield reliable and tight lower bounds in various continuous control environments.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 35 of its 51 references that are in the map. First posted on arXiv 2 Mar 2023.

Builds on

35 of 51 references.

References · 35

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  7. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  8. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  9. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  10. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  11. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  12. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
23 more
  1. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  2. 2016Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithmcited by 507
  3. 2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 422
  4. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 371
  5. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 370
  6. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 366
  7. 2017On Kernelized Multi-armed Banditscited by 352
  8. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  9. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
  10. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  11. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 188
  12. 2020On Information Gain and Regret Bounds in Gaussian Process Banditscited by 158
  13. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  14. 2019Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processescited by 119
  15. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  16. 2015An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learningcited by 103
  17. 2020Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planningcited by 86
  18. 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
  19. 2017Consistent On-Line Off-Policy Evaluationcited by 51
  20. 2020Accountable Off-Policy Evaluation With Kernel Bellman Statisticscited by 36
  21. 2021Deeply-Debiased Off-Policy Interval Estimationcited by 36
  22. 2020Statistical Bootstrapping for Uncertainty Estimation in Off-Policy Evaluationcited by 28
  23. 2021Non-asymptotic Confidence Intervals of Off-policy Evaluation: Primal and Dual Boundscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 5; 2025: 3; 2026: 2.

13 papers.

Sort:

By area:

  1. 2023Optimistic Active Exploration of Dynamical Systemscited by 32
  2. 2024ActSafe: Active Exploration with Safety Constraints for Reinforcement Learningcited by 23
  3. 2023Efficient Exploration in Continuous-time Model-based Reinforcement Learningcited by 22
  4. 2024When to Sense and Control? A Time-adaptive Approach for Continuous-Time RLcited by 9
  5. 2024Bridging the Sim-to-Real Gap with Bayesian Inferencecited by 8
  6. 2025SOMBRL: Scalable and Optimistic Model-Based RLcited by 7
  7. 2023Data-Efficient Task Generalization via Probabilistic Model-based Meta Reinforcement Learningcited by 6
  8. 2024NeoRL: Efficient Exploration for Nonepisodic RLcited by 4
  9. 2025Sample-efficient and Scalable Exploration in Continuous-Time RLcited by 3
  10. 2026Model-Based Reinforcement Learning for Control under Time-Varying Dynamicscited by 1
  11. 2024Active Learning of Dynamics Using Prior Domain Knowledge in the Sampling Processcited by 0
  12. 2025Simulation Priors for Data-Efficient Deep Learningcited by 0
1 more
  1. 2026UBP2: Uncertainty-Balanced Preference Planning for Efficient Preference-based Reinforcement Learningcited by 0