mlsanityData to 14 Oct 2026

Contextualized Hybrid Ensemble Q-learning: Learning Fast with Control Priors

arXiv 2406.19768 ↗First posted 28 Jun 2024cs.LGcited by 5

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Combining Reinforcement Learning (RL) with a prior controller can yield the best out of two worlds: RL can solve complex nonlinear problems, while the control prior ensures safer exploration and speeds up training. Prior work largely blends both components with a fixed weight, neglecting that the RL agent's performance varies with the training progress and across regions in the state space. Therefore, we advocate for an adaptive strategy that dynamically adjusts the weighting based on the RL agent's current capabilities. We propose a new adaptive hybrid RL algorithm, Contextualized Hybrid Ensemble Q-learning (CHEQ). CHEQ combines three key ingredients: (i) a time-invariant formulation of the adaptive hybrid RL problem treating the adaptive weight as a context variable, (ii) a weight adaption mechanism based on the parametric uncertainty of a critic ensemble, and (iii) ensemble-based acceleration for data-efficient RL. Evaluating CHEQ on a car racing task reveals substantially stronger data efficiency, exploration safety, and transferability to unknown scenarios than state-of-the-art adaptive hybrid RL methods.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 25 of its 37 references that are in the map. First posted on arXiv 28 Jun 2024.

Builds on

25 of 37 references.

References · 25

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  5. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  6. 2016Deep Exploration via Bootstrapped DQNcited by 527
  7. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  8. 2018Residual Reinforcement Learning for Robot Controlcited by 274
  9. 2018Residual Policy Learningcited by 164
  10. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 152
  11. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
  12. 2021Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Interventioncited by 95
13 more
  1. 2019Deep Reinforcement Learning for Industrial Insertion Tasks with Visual Inputs and Natural Rewardscited by 64
  2. 2019Control Regularization for Reduced Variance Reinforcement Learningcited by 37
  3. 2020Learning to Play Table Tennis From Scratch using Muscular Robotscited by 35
  4. 2022Sample Efficient Deep Reinforcement Learning via Uncertainty Estimationcited by 27
  5. 2021Bayesian Controller Fusion: Leveraging Control Priors in Deep Reinforcement Learning for Roboticscited by 20
  6. 2020Reinforcement Learning with Uncertainty Estimation for Tactical Decision-Making in Intersectionscited by 15
  7. 2020Tactical Decision-Making in Autonomous Driving by Reinforcement Learning with Uncertainty Estimationcited by 14
  8. 2019Residual Reactive Navigation: Combining Classical and Learned Navigation Strategies For Deployment in Unknown Environmentscited by 11
  9. 2021Residual Feedback Learning for Contact-Rich Manipulation Tasks with Uncertaintycited by 10
  10. 2023Adaptive Ensemble Q-learning: Minimizing Estimation Bias via Error Feedbackcited by 9
  11. 2024RESPRECT: Speeding-up Multi-fingered Grasping with Residual Reinforcement Learningcited by 4
  12. 2024Enhancing Reinforcement Learning Agents with Local Guidescited by 4
  13. 2022Residual Policy Learning for Powertrain Controlcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 2.

5 papers.

  1. 2026Rainbow-DemoRL: Combining Improvements in Demonstration-Augmented Reinforcement Learningcited by 2
  2. 2024Stepping Out of the Shadows: Reinforcement Learning in Shadow Modecited by 1
  3. 2025CHEQ-ing the Box: Safe Variable Impedance Learning for Robotic Polishingcited by 1
  4. 2025Partially Observable Residual Reinforcement Learning for PV-Inverter-Based Voltage Control in Distribution Gridscited by 0
  5. 2026FLEET: Token-Based Feature Extraction for Event Camera-based Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Multi-Timescale Ensemble Q-learning for Markov Decision Process Policy Optimization
  2. 2024ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
  3. 2025Hybrid Action Based Reinforcement Learning for Multi-Objective Compatible Autonomous Driving
  4. 2024Towards Adapting Reinforcement Learning Agents to New Tasks: Insights from Q-Values
  5. 2025Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Network
  6. 2026In-Context Reinforcement Learning through Bayesian Fusion of Context and Value Prior
  7. 2024SPQR: Controlling Q-ensemble Independence with Spiked Random Model for Reinforcement Learning
  8. 2025RLAE: Reinforcement Learning-Assisted Ensemble for LLMs