Data to 14 Oct 2026

Small batch deep reinforcement learning

arXiv 2310.03882 ↗First posted 5 Oct 2023cs.LG cs.AIcited by 36

Not on the map. Why?

Cited by 32 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In value-based deep reinforcement learning with replay memories, the batch size parameter specifies how many transitions to sample for each gradient update. Although critical to the learning process, this value is typically not adjusted when proposing new algorithms. In this work we present a broad empirical study that suggests em reducing the batch size can result in a number of significant performance gains; this is surprising, as the general tendency when training neural networks is towards larger batch sizes for improved performance. We complement our experimental findings with a set of empirical analyses towards better understanding this phenomenon.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Builds on 40 of its 60 references that are in the map. First posted on arXiv 5 Oct 2023.

Builds on

40 of 60 references.

References · 40

2014Adam: A Method for Stochastic Optimizationcited by 37,240
2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
2017A Distributional Perspective on Reinforcement Learningcited by 765
2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
2019Model-Based Reinforcement Learning for Ataricited by 502
2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 318

28 more

2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
2017Noisy Networks for Explorationcited by 266
2022The Primacy Bias in Deep Reinforcement Learningcited by 247
2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 193
2020Revisiting Fundamentals of Experience Replaycited by 175
2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 174
2023Understanding plasticity in neural networkscited by 174
2017Parameter Space Noise for Explorationcited by 161
2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 122
2022Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learningcited by 122
2019When to use parametric models in reinforcement learning?cited by 118
2018Revisiting Small Batch Training for Deep Neural Networkscited by 102
2017Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Controlcited by 93
2020Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Researchcited by 78
2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularizationcited by 66
2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 65
2020The Value-Improvement Path: Towards Better Representations for Reinforcement Learningcited by 60
2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 58
2023Proto-Value Networks: Scaling Representation Learning with Auxiliary Taskscited by 42
2018Accelerated Methods for Deep Reinforcement Learningcited by 40
2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
2021On Bonus-Based Exploration Methods in the Arcade Learning Environmentcited by 37
2021Batch size-invariance for policy optimizationcited by 35
2018On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descentcited by 25
2021Large Batch Experience Replaycited by 14
2021Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddingscited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 15; 2025: 13; 2026: 8.

36 papers.

Sort:

By area:

2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 108
2024Mixtures of Experts Unlock Parameter Scaling for Deep RLcited by 78
2024Simplifying Deep Temporal Difference Learningcited by 78
20251000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilitiescited by 45
2024In value-based deep reinforcement learning, a pruned network is a good networkcited by 39
2024Plasticity Loss in Deep Reinforcement Learning: A Surveycited by 24
2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learningcited by 23
2025Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churncited by 22
2024On the consistency of hyper-parameter selection in value-based deep reinforcement learningcited by 21
2024Mixture of Experts in a Mixture of RL settingscited by 21
2024Neuroplastic Expansion in Deep Reinforcement Learningcited by 18
2024Continuous Control with Coarse-to-fine Reinforcement Learningcited by 17

24 more

2024Can Learned Optimization Make Reinforcement Learning Less Difficult?cited by 14
2024Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RLcited by 14
2025Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learningcited by 14
2025The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networkscited by 13
2025Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agentscited by 12
2024Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PCcited by 7
2025Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Datacited by 7
2025The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learningcited by 6
2025Compute-Optimal Scaling for Value-Based Deep RLcited by 6
2025How Should We Meta-Learn Reinforcement Learning Algorithms?cited by 5
2026Stable Deep Reinforcement Learning via Isotropic Gaussian Representationscited by 5
2024ARLBench: Flexible and Efficient Benchmarking for Hyperparameter Optimization in Reinforcement Learningcited by 3
2025The Formalism-Implementation Gap in Reinforcement Learning Researchcited by 3
2024Higher Replay Ratio Empowers Sample-Efficient Multi-Agent Reinforcement Learningcited by 2
2025Plasticine: Accelerating Research in Plasticity-Motivated Deep Reinforcement Learningcited by 2
2026Do Neural Networks Lose Plasticity in a Gradually Changing World?cited by 2
2026The Rank and Gradient Lost in Non-stationarity: Sample Weight Decay for Mitigating Plasticity Loss in Reinforcement Learningcited by 2
2026Naive Prompt Optimization: Rethinking the Need for Complex Prompt Searchcited by 2
2024Deep deterministic policy gradient with symmetric data augmentation for lateral attitude tracking control of a fixed-wing aircraftcited by 1
2025Recurrent Off-Policy Deep Reinforcement Learning Doesn't Have to be Slowcited by 0
2026Rethinking Plasticity in Deep Reinforcement Learningcited by 0
2026Scalable Reinforcement Learning via Adaptive Batch Scalingcited by 0
2026Representation Learning Enables Scalable Multitask Deep Reinforcement Learningcited by 0
2026Mixture of GFlowNetscited by 0