mlsanityData to 7 Oct 2026

Dense and Diverse Goal Coverage in Multi Goal Reinforcement Learning

arXiv 2510.25311 ↗First posted 29 Oct 2025cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Reinforcement Learning algorithms are primarily focused on learning a policy that maximizes expected return. As a result, the learned policy can exploit one or few reward sources. However, in many natural situations, it is desirable to learn a policy that induces a dispersed marginal state distribution over rewarding states, while maximizing the expected return which is typically tied to reaching a goal state. This aspect remains relatively unexplored. Existing techniques based on entropy regularization and intrinsic rewards use stochasticity for encouraging exploration to find an optimal policy which may not necessarily lead to dispersed marginal state distribution over rewarding states. Other RL algorithms which match a target distribution assume the latter to be available apriori. This may be infeasible in large scale systems where enumeration of all states is not possible and a state is determined to be a goal state only upon reaching it. We formalize the problem of maximizing the expected return while uniformly visiting the goal states as Multi Goal RL in which an oracle classifier over the state space determines the goal states. We propose a novel algorithm that learns a high-return policy mixture with marginal state distribution dispersed over the set of goal states. Our algorithm is based on optimizing a custom RL reward which is computed - based on the current policy mixture - at each iteration for a set of sampled trajectories. The latter are used via an offline RL algorithm to update the policy mixture. We prove performance guarantees for our algorithm, showing efficient convergence bounds for optimizing a natural objective which captures the expected return as well as the dispersion of the marginal state distribution over the goal states. We design and perform experiments on synthetic MDPs and standard RL environments to evaluate the effectiveness of our algorithm.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 35 of its 47 references that are in the map. First posted on arXiv 29 Oct 2025.

Builds on

35 of 47 references.

References · 35

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  6. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  7. 2017Hindsight Experience Replaycited by 1,096
  8. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  9. 2018Exploration by Random Network Distillationcited by 966
  10. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  11. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  12. 2021Flow Network based Generative Models for Non-Iterative Diverse Candidate Generationcited by 384
23 more
  1. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
  2. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
  3. 2018Maximum a Posteriori Policy Optimisationcited by 318
  4. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  5. 2022Trajectory balance: Improved credit assignment in GFlowNetscited by 244
  6. 2019Batch Policy Learning under Constraintscited by 242
  7. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  8. 2018Provably Efficient Maximum Entropy Explorationcited by 240
  9. 2019Efficient Exploration via State Marginal Matchingcited by 186
  10. 2018Understanding the impact of entropy on policy optimizationcited by 171
  11. 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 145
  12. 2019Planning with Goal-Conditioned Policiescited by 144
  13. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  14. 2020Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learningcited by 100
  15. 2020C-Learning: Learning to Achieve Goals via Recursive Classificationcited by 99
  16. 2017Surprise-Based Intrinsic Motivation for Deep Reinforcement Learningcited by 83
  17. 2019Maximum Entropy-Regularized Multi-Goal Reinforcement Learningcited by 39
  18. 2021A Max-Min Entropy Framework for Reinforcement Learningcited by 28
  19. 2023Planning Goals for Explorationcited by 28
  20. 2025Offline Goal-conditioned Reinforcement Learning with Quasimetric Representationscited by 20
  21. 2023Searching for High-Value Molecules Using Reinforcement Learning and Transformerscited by 18
  22. 2024Accelerating Goal-Conditioned RL Algorithms and Researchcited by 18
  23. 2025When Maximum Entropy Misleads Policy Optimizationcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Goal-conditioned Offline Reinforcement Learning through State Space Partitioning
  2. 2023Reinforcement Learning with General Utilities: Simpler Variance Reduction and Large State-Action Space
  3. 2025Optimal Single-Policy Sample Complexity and Transient Coverage for Average-Reward Offline RL
  4. 2026Learning from the Near Future: Temporal Self-Distillation for RLVR
  5. 2026Reward-free Pretraining for Reinforcement Learning via Occupancy Coverage Maximization
  6. 2020Exploration by Maximizing R'enyi Entropy for Reward-Free RL Framework
  7. 2023Learning to Reach Goals via Diffusion
  8. 2021On Multi-objective Policy Optimization as a Tool for Reinforcement Learning: Case Studies in Offline RL and Finetuning