mlsanityData to 14 Oct 2026

Enhancing Diversity in Parallel Agents: A Maximum State Entropy Exploration Story

arXiv 2505.01336 ↗First posted 2 May 2025cs.LGcited by 4

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Parallel data collection has redefined Reinforcement Learning (RL), unlocking unprecedented efficiency and powering breakthroughs in large-scale real-world applications. In this paradigm, N identical agents operate in N replicas of an environment simulator, accelerating data collection by a factor of N. A critical question arises: textitDoes specializing the policies of the parallel agents hold the key to surpass the N factor acceleration? In this paper, we introduce a novel learning framework that maximizes the entropy of collected data in a parallel setting. Our approach carefully balances the entropy of individual agents with inter-agent diversity, effectively minimizing redundancies. The latter idea is implemented with a centralized policy gradient method, which shows promise when evaluated empirically against systems of identical agents, as well as synergy with batch RL techniques that can exploit data diversity. Finally, we provide an original concentration analysis that shows faster rates for specialized parallel sampling distributions, which supports our methodology and may be of independent interest.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 45 of its 68 references that are in the map. First posted on arXiv 2 May 2025.

Builds on

45 of 68 references.

References · 45

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  2. 2015Trust Region Policy Optimizationcited by 2,580
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  4. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  5. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  6. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 512
  7. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 313
  8. 2016Variational Intrinsic Controlcited by 267
  9. 2018Provably Efficient Maximum Entropy Explorationcited by 242
  10. 2019Efficient Exploration via State Marginal Matchingcited by 188
  11. 2021Reinforcement Learning with Prototypical Representationscited by 179
  12. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 174
33 more
  1. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 141
  2. 2019Fast Task Inference with Variational Intrinsic Successor Featurescited by 133
  3. 2020Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skillscited by 122
  4. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 108
  5. 2021Celebrating Diversity in Shared Multi-Agent Reinforcement Learningcited by 98
  6. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 96
  7. 2019Influence-Based Multi-Agent Explorationcited by 75
  8. 2021Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordinationcited by 70
  9. 2018Policy Optimization via Importance Samplingcited by 47
  10. 2021MADE: Exploration via Maximizing Deviation from Explored Regionscited by 40
  11. 2022Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimalitycited by 37
  12. 2022The Importance of Non-Markovianity in Maximum State Entropy Explorationcited by 35
  13. 2023Fast Rates for Maximum Entropy Explorationcited by 33
  14. 2017Efficient Parallel Methods for Deep Reinforcement Learningcited by 32
  15. 2021Geometric Entropic Explorationcited by 32
  16. 2022Challenging Common Assumptions in Convex Reinforcement Learningcited by 30
  17. 2019Active Exploration in Markov Decision Processescited by 27
  18. 2022An Analysis of Ensemble Samplingcited by 24
  19. 2023Maximum State Entropy Exploration using Predecessor and Successor Representationscited by 23
  20. 2021Wasserstein Unsupervised Reinforcement Learningcited by 22
  21. 2023Accelerating Reinforcement Learning with Value-Conditional State Entropy Explorationcited by 22
  22. 2020Population-Guided Parallel Policy Search for Reinforcement Learningcited by 21
  23. 2019An Intrinsically-Motivated Approach for Learning Highly Exploring and Fast Mixing Policiescited by 20
  24. 2020Active Model Estimation in Markov Decision Processescited by 20
  25. 2023Explore to Generalize in Zero-Shot RLcited by 20
  26. 2018Coordinated Exploration in Concurrent Reinforcement Learningcited by 17
  27. 2021Unsupervised Reinforcement Learning in Multiple Environmentscited by 17
  28. 2022k-Means Maximum Entropy Explorationcited by 11
  29. 2018Scalable Coordinated Exploration in Concurrent Reinforcement Learningcited by 10
  30. 2023Self-Motivated Multi-Agent Explorationcited by 7
  31. 2023No Prior Mask: Eliminate Redundant Action for Deep Reinforcement Learningcited by 6
  32. 2024How to Explore with Belief: State Entropy Maximization in POMDPscited by 6
  33. 2024The Limits of Pure Exploration in POMDPs: When the Observation Entropy is Enoughcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025From Parameters to Behaviors: Unsupervised Compression of the Policy Spacecited by 4
  2. 2026K-Myriad: Jump-starting reinforcement learning with unsupervised parallel agentscited by 0
  3. 2026Agent Economics: An Entropy-Controlled Pluralistic Alignment Framework for Preventing Artificial Hivemind in Autonomous Agentscited by 0
  4. 2026Marginal Coverage Credit Reduces Redundant Exploration in Parallel State-Entropy Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Rethinking Policy Diversity in Ensemble Policy Gradient in Large-Scale Reinforcement Learning
  2. 2025The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks
  3. 2025Scalable Multi-Agent Offline Reinforcement Learning and the Role of Information
  4. 2025Efficiency of Parallel and Restart Exploration Strategies in Model Free Stochastic Simulations
  5. 2024Novelty-Guided Data Reuse for Efficient and Diversified Multi-Agent Reinforcement Learning
  6. 2024The Curse of Diversity in Ensemble-Based Exploration
  7. 2024HyperMARL: Adaptive Hypernetworks for Multi-Agent RL
  8. 2024Measuring Policy Distance for Multi-Agent Reinforcement Learning