mlsanityData to 14 Oct 2026

Smart Sampling: Self-Attention and Bootstrapping for Improved Ensembled Q-Learning

arXiv 2405.08252 ↗First posted 14 May 2024cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

We present a novel method aimed at enhancing the sample efficiency of ensemble Q learning. Our proposed approach integrates multi-head self-attention into the ensembled Q networks while bootstrapping the state-action pairs ingested by the ensemble. This not only results in performance improvements over the original REDQ (Chen et al. 2021) and its variant DroQ (Hi-raoka et al. 2022), thereby enhancing Q predictions, but also effectively reduces both the average normalized bias and standard deviation of normalized bias within Q-function ensembles. Importantly, our method also performs well even in scenarios with a low update-to-data (UTD) ratio. Notably, the implementation of our proposed method is straightforward, requiring minimal modifications to the base model.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 28 references that are in the map. First posted on arXiv 14 May 2024.

Builds on

22 of 28 references.

References · 22

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  5. 2016Layer Normalizationcited by 5,669
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  7. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  8. 2015Trust Region Policy Optimizationcited by 2,580
  9. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  10. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  11. 2018Learning Dexterous In-Hand Manipulationcited by 687
  12. 2016Deep Exploration via Bootstrapped DQNcited by 527
10 more
  1. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  2. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 152
  3. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 114
  4. 2019Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shiftcited by 47
  5. 2021UPDeT: Universal Multi-agent Reinforcement Learning via Policy Decoupling with Transformerscited by 39
  6. 2020Bidirectional Model-based Policy Optimizationcited by 38
  7. 2019Guided Meta-Policy Searchcited by 26
  8. 2020Model-based Policy Optimization with Unsupervised Model Adaptationcited by 16
  9. 2019Transformer Based Reinforcement Learning For Gamescited by 5
  10. 2022Transformer-based Value Function Decomposition for Cooperative Multi-agent Reinforcement Learning in StarCraftcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024SPQR: Controlling Q-ensemble Independence with Spiked Random Model for Reinforcement Learning
  2. 2026Revisiting TD Target Aggregation under Uncertainty in Q-Learning
  3. 2026Revisiting Overestimation Bias Problem of Q-learning: Settling Large Discrete Action Space via Action Intersection
  4. 2024MinMaxMin Q-learning
  5. 2025Dual Ensembled Multiagent Q-Learning with Hypernet Regularizer
  6. 2026SageBwd: A Trainable Low-bit Attention
  7. 2025GB-DQN: Gradient Boosted DQN Models for Non-stationary Reinforcement Learning
  8. 2025Enhancing Q-Value Updates in Deep Q-Learning via Successor-State Prediction