Conservative Query and Adaptive Regularization for Offline RL Under Uncertainty Estimation
On the map: Offline RL and imitation learning from data → Offline RL algorithms
Abstract (arXiv)
Offline reinforcement learning (RL) aims to learn an effective policy from a static dataset, but its performance is fundamentally limited by dataset coverage. Action preference queries leverage expert feedback without additional environment interaction, enabling policy improvement during offline training. However, existing methods still face two key challenges: selecting informative preference queries and effectively exploiting the collected feedback. Current approaches typically rely only on the distance between policy actions and dataset actions for query selection, while enforcing fixed constraints that keep the policy close to queried preferences. Such strategies often lead to unstable policy updates and integrate poorly with value regularization. To address these limitations, we propose Conservative Query and Adaptive Regularization under Uncertainty Estimation, a lightweight framework that jointly improves preference querying and preference exploitation. Specifically, we employ a Morse network to estimate the uncertainty of policy actions with respect to the offline dataset. Based on this uncertainty, we introduce a conservative query strategy that selectively queries actions near the dataset to preserve Bellman-update stability, together with an uncertainty-aware adaptive regularization scheme that dynamically adjusts data-level constraints during policy optimization. We integrate our framework with CQL and evaluate it extensively on the D4RL benchmark. Experimental results demonstrate superior or competitive performance across a wide range of tasks.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 30 of its 44 references that are in the map. First posted on arXiv 21 Jul 2026.
Builds on
30 of 44 references.
Closest ancestors · 18
Same direction, newest first
- 2024Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL25 Dec
- 2024Offline Reinforcement Learning with Behavioral Supervisor Tuning25 Apr
- 2023Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learning27 Oct
- 2023Boosting Offline Reinforcement Learning with Action Preference Query6 Jun
- 2023Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcare2 May
- 2022Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Size20 Nov
- 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
- 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
- 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
- 2021Offline RL Without Off-Policy Evaluation16 Jun
- 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
- 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
6 more
- 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
- 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
- 2020MOReL : Model-Based Offline Reinforcement Learning12 May
- 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
- 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
- 2019Behavior Regularized Offline Reinforcement Learning26 Nov
Other references · 12
- 2017Deep reinforcement learning from human preferencescited by 3,846
- 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
- 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
- 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
- 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
- 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
- 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
- 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
- 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
- 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
- 2022Improving TD3-BC: Relaxed Policy Constraint for Offline Learning and Stable Online Fine-Tuningcited by 20
- 2023Morse Neural Networks for Uncertainty Quantificationcited by 4
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2023Policy Regularization with Dataset Constraint for Offline Reinforcement Learning
- 2025Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL
- 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangian
- 2025Mildly Conservative Regularized Evaluation for Offline Reinforcement Learning
- 2023Iteratively Refined Behavior Regularization for Offline Reinforcement Learning
- 2026Robust Regularized Policy Iteration under Transition Uncertainty
- 2024Policy-regularized Offline Multi-objective Reinforcement Learning
- 2020Expert-Supervised Reinforcement Learning for Offline Policy Learning and Evaluation