mlsanityData to 7 Oct 2026

Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Size

arXiv 2211.11092 ↗First posted 20 Nov 2022cs.LG cs.AI cs.NEcited by 20

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Training large neural networks is known to be time-consuming, with the learning duration taking days or even weeks. To address this problem, large-batch optimization was introduced. This approach demonstrated that scaling mini-batch sizes with appropriate learning rate adjustments can speed up the training process by orders of magnitude. While long training time was not typically a major issue for model-free deep offline RL algorithms, recently introduced Q-ensemble methods achieving state-of-the-art performance made this issue more relevant, notably extending the training duration. In this work, we demonstrate how this class of methods can benefit from large-batch optimization, which is commonly overlooked by the deep offline RL community. We show that scaling the mini-batch size and naively adjusting the learning rate allows for (1) a reduced size of the Q-ensemble, (2) stronger penalization of out-of-distribution actions, and (3) improved convergence time, effectively shortening training duration by 3-4x times on average.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 45 of its 54 references that are in the map. First posted on arXiv 20 Nov 2022.

Builds on

45 of 54 references.

Closest ancestors · 15

Same direction, newest first

  1. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters27 May
  2. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  3. 2021d3rlpy: An Offline Deep Reinforcement Learning Library6 Nov
  4. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  5. 2021Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters8 Oct
  6. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  7. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
  8. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  9. 2021Offline Reinforcement Learning as Anti-Exploration11 Jun
  10. 2021Personalization for Web-based Services using Offline Reinforcement Learning10 Feb
  11. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
  12. 2020Critic Regularized Regression26 Jun
3 more
  1. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 30

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2016Layer Normalizationcited by 5,579
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  6. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  7. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  8. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  9. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  10. 2017Large Batch Training of Convolutional Networkscited by 529
  11. 2016Deep Exploration via Bootstrapped DQNcited by 518
  12. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
18 more
  1. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  2. 2018An Empirical Model of Large-Batch Trainingcited by 268
  3. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  4. 2018Distributed Prioritized Experience Replaycited by 260
  5. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
  6. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
  7. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 150
  8. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 108
  9. 2019Better Exploration with Optimistic Actor-Criticcited by 88
  10. 2022Can Wikipedia Help Offline Reinforcement Learning?cited by 76
  11. 2021DeepThermal: Combustion Optimization for Thermal Power Generating Units Using Offline Reinforcement Learningcited by 47
  12. 2018Accelerated Methods for Deep Reinforcement Learningcited by 40
  13. 2020Bidirectional Model-based Policy Optimizationcited by 38
  14. 2021LASER: Learning a Latent Action Space for Efficient Reinforcement Learningcited by 32
  15. 2017Shallow Updates for Deep Reinforcement Learningcited by 14
  16. 2022Reducing Variance in Temporal-Difference Value Estimation via Ensemble of Deep Networkscited by 13
  17. 2022DNS: Determinantal Point Process Based Neural Network Sampler for Ensemble Reinforcement Learningcited by 8
  18. 2018Distributed Deep Reinforcement Learning: Learn how to play Atari games in 21 minutescited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 7; 2025: 3; 2026: 4.

20 papers.

Sort:

By area:

  1. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 121
  2. 2023XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAXcited by 54
  3. 2023Anti-Exploration by Random Network Distillationcited by 49
  4. 2023Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RLcited by 20
  5. 2025Reinforcement Fine-Tuning of Flow-Matching Policies for Vision-Language-Action Modelscited by 7
  6. 2023Decoupled Prioritized Resampling for Offline RLcited by 6
  7. 2025Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RLcited by 5
  8. 2023R^3: On-device Real-Time Deep Reinforcement Learning for Autonomous Roboticscited by 4
  9. 2024Is Value Functions Estimation with Classification Plug-and-play for Offline Reinforcement Learning?cited by 4
  10. 2024The Role of Deep Learning Regularizations on Actors in Offline RLcited by 2
  11. 2024NetworkGym: Reinforcement Learning Environments for Multi-Access Traffic Management in Network Simulationcited by 2
  12. 2024Dynamic Non-Prehensile Object Transport via Model-Predictive Reinforcement Learningcited by 2
8 more
  1. 2024Dataset Clustering for Improved Offline Policy Learningcited by 1
  2. 2024Robustness Evaluation of Offline Reinforcement Learning for Robot Control Against Action Perturbationscited by 1
  3. 2026Less is More: Clustered Cross-Covariance Control for Offline RLcited by 1
  4. 2026AdamO: A Collapse-Suppressed Optimizer for Offline RLcited by 1
  5. 2024The Virtues of Pessimism in Inverse Reinforcement Learningcited by 0
  6. 2025Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategiescited by 0
  7. 2026A Pontryagin Method of Model-based Reinforcement Learning via Hamiltonian Actor-Criticcited by 0
  8. 2026Conservative Query and Adaptive Regularization for Offline RL Under Uncertainty Estimationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensembles
  2. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizes
  3. 2025Scaling Offline RL via Efficient and Expressive Shortcut Models
  4. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trials
  5. 2026Fast Catch-Up, Late Switching: Optimal Batch Size Scheduling via Functional Scaling Laws
  6. 2025Pretraining a Shared Q-Network for Data-Efficient Offline Reinforcement Learning
  7. 2023Pre-training with Synthetic Data Helps Offline Reinforcement Learning
  8. 2024SPQR: Controlling Q-ensemble Independence with Spiked Random Model for Reinforcement Learning