mlsanityData to 13 Oct 2026

Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RL

arXiv 2410.01930 ↗First posted 2 Oct 2024cs.LG cs.AIcited by 14

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

The use of deep neural networks in reinforcement learning (RL) often suffers from performance degradation as model size increases. While soft mixtures of experts (SoftMoEs) have recently shown promise in mitigating this issue for online RL, the reasons behind their effectiveness remain largely unknown. In this work we provide an in-depth analysis identifying the key factors driving this performance gain. We discover the surprising result that tokenizing the encoder output, rather than the use of multiple experts, is what is behind the efficacy of SoftMoEs. Indeed, we demonstrate that even with an appropriately scaled single expert, we are able to maintain the performance gains, largely thanks to tokenization.

Key facts. Cited by 14 papers in the map (data through 13 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 43 of its 64 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

43 of 64 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Mixture of Experts in a Mixture of RL settings26 Jun
  2. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RL6 Mar
  3. 2024Disentangling the Causes of Plasticity Loss in Neural Networks29 Feb
  4. 2024In value-based deep reinforcement learning, a pruned network is a good network19 Feb
  5. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RL13 Feb

Other references · 38

  1. 2017Proximal Policy Optimization Algorithmscited by 14,592
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,656
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,463
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,044
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,618
  6. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 871
  7. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 790
  8. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 777
  9. 2017A Distributional Perspective on Reinforcement Learningcited by 745
  10. 2022Mixture-of-Experts with Expert Choice Routingcited by 619
  11. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 479
  12. 2019CondConv: Conditionally Parameterized Convolutions for Efficient Inferencecited by 273
26 more
  1. 2023From Sparse to Soft Mixtures of Expertscited by 247
  2. 2022The Primacy Bias in Deep Reinforcement Learningcited by 243
  3. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 207
  4. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 173
  5. 2023Understanding plasticity in neural networkscited by 171
  6. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
  7. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  8. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 124
  9. 2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 121
  10. 2019When to use parametric models in reinforcement learning?cited by 118
  11. 2021DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learningcited by 115
  12. 2020Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Researchcited by 78
  13. 2020Transient Non-Stationarity and Generalisation in Deep Reinforcement Learningcited by 75
  14. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizescited by 72
  15. 2022The State of Sparse Training in Deep Reinforcement Learningcited by 61
  16. 2020Scalable Transfer Learning with Expert Modelscited by 49
  17. 2021Dynamic Sparse Training for Deep Reinforcement Learningcited by 49
  18. 2024Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learningcited by 49
  19. 2023TaskExpert: Dynamically Assembling Multi-Task Representations with Memorial Mixture-of-Expertscited by 45
  20. 2023Proto-Value Networks: Scaling Representation Learning with Auxiliary Taskscited by 42
  21. 2024Slow and Steady Wins the Race: Maintaining Plasticity with Hare and Tortoise Networkscited by 41
  22. 2023Small batch deep reinforcement learningcited by 35
  23. 2023Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stagescited by 27
  24. 2023Directions of Curvature as an Explanation for Loss of Plasticitycited by 26
  25. 2020Biased Mixtures Of Experts: Enabling Computer Vision Inference Under Data Transfer Limitationscited by 20
  26. 2024Beyond Parameter Count: Implicit Bias in Soft Mixture of Expertscited by 1

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 7.

14 papers.

Sort:

By area:

  1. 2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learningcited by 23
  2. 2024Neuroplastic Expansion in Deep Reinforcement Learningcited by 18
  3. 2025Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learningcited by 14
  4. 2025Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agentscited by 12
  5. 2025Impoola: The Power of Average Pooling for Image-Based Deep Reinforcement Learningcited by 6
  6. 2025The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learningcited by 6
  7. 2026Stable Deep Reinforcement Learning via Isotropic Gaussian Representationscited by 5
  8. 2025Mind the GAP! The Challenges of Scale in Pixel-based Deep Reinforcement Learningcited by 4
  9. 2026SPHERE: Mitigating the Loss of Spectral Plasticity in Mixture-of-Experts for Deep Reinforcement Learningcited by 2
  10. 2026Temporal Recurrence Favors Fewer Layerscited by 1
  11. 2026Sparse Masked Attention Policies for Reliable Generalizationcited by 0
  12. 2026Higher Resolution, Better Generalization: Unlocking Visual Scaling in Deep Reinforcement Learningcited by 0
2 more
  1. 2026Representation Learning Enables Scalable Multitask Deep Reinforcement Learningcited by 0
  2. 2026V-Simba: Unleashing the Architectural Potential of RL in Visual Continuous Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024The Impact of Quantization and Pruning on Deep Reinforcement Learning Models
  2. 2021Single-Shot Pruning for Offline Reinforcement Learning
  3. 2025Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs
  4. 2026Which Tokens Matter? Adaptive Token Selection for RLVR with the Relative Surprisal Index
  5. 2025Rethinking RL Evaluation: Can Benchmarks Truly Reveal Failures of RL Methods?
  6. 2026Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training
  7. 2025Towards Stable and Effective Reinforcement Learning for Mixture-of-Experts
  8. 2026SoftMoE: Soft Differentiable Routing for Mixture-of-Experts in LLMs