mlsanityData to 7 Oct 2026

Smart Sampling: Self-Attention and Bootstrapping for Improved Ensembled Q-Learning

arXiv 2405.08252 ↗First posted 14 May 2024cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

We present a novel method aimed at enhancing the sample efficiency of ensemble Q learning. Our proposed approach integrates multi-head self-attention into the ensembled Q networks while bootstrapping the state-action pairs ingested by the ensemble. This not only results in performance improvements over the original REDQ (Chen et al. 2021) and its variant DroQ (Hi-raoka et al. 2022), thereby enhancing Q predictions, but also effectively reduces both the average normalized bias and standard deviation of normalized bias within Q-function ensembles. Importantly, our method also performs well even in scenarios with a low update-to-data (UTD) ratio. Notably, the implementation of our proposed method is straightforward, requiring minimal modifications to the base model.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 22 of its 28 references that are in the map. First posted on arXiv 14 May 2024.

Builds on

22 of 28 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  5. 2016Layer Normalizationcited by 5,579
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  7. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  8. 2015Trust Region Policy Optimizationcited by 2,503
  9. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  10. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  11. 2018Learning Dexterous In-Hand Manipulationcited by 643
  12. 2016Deep Exploration via Bootstrapped DQNcited by 518
10 more
  1. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  2. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 150
  3. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 108
  4. 2019Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shiftcited by 47
  5. 2021UPDeT: Universal Multi-agent Reinforcement Learning via Policy Decoupling with Transformerscited by 39
  6. 2020Bidirectional Model-based Policy Optimizationcited by 38
  7. 2019Guided Meta-Policy Searchcited by 26
  8. 2020Model-based Policy Optimization with Unsupervised Model Adaptationcited by 16
  9. 2019Transformer Based Reinforcement Learning For Gamescited by 5
  10. 2022Transformer-based Value Function Decomposition for Cooperative Multi-agent Reinforcement Learning in StarCraftcited by 5

Built on by

Not cited yet.