mlsanityData to 14 Oct 2026

MIPS: Mitigating Mode Collapse in Reinforcement Learning for Large Outcome Spaces

arXiv 2610.19457 ↗First posted 14 Oct 2026cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations also fits: RL theory and sample complexity

Abstract (arXiv)

Expected-reward maximization in reinforcement learning concentrates probability mass on a few outcomes, even in tasks that require many distinct, high-quality solutions, a failure known as mode collapse. Inverse Probability Scaling (IPS) counters this in group-based policy gradients such as GRPO by dividing each reward by the outcome's probability, estimated from its frequency within the sampled group. When the outcome space is large relative to the group, outcomes rarely repeat and these estimates become uninformative. Since an outcome can be constructed along many trajectories, its probability is a sum over all of them, and computing it directly is intractable. We propose Marginal IPS (MIPS), which performs marginal evaluation by combining the sampled trajectory with a learned model of the trajectories that lead to each outcome. On Hypergrids, where the exact sampling distribution is computable, MIPS stays near the reward-distribution target as the outcome space grows relative to the rollout group, while frequency-based IPS degrades. In molecular synthesis and phylogenetic inference, MIPS sustains diverse, high-quality sampling where GRPO and frequency-based IPS collapse, showing that MIPS serves as a reliable solution to mode collapse in RL.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 20 of its 22 references that are in the map. First posted on arXiv 14 Oct 2026.

Builds on

20 of 22 references.

Closest ancestors · 1

Same direction, newest first

  1. 2026Expected Return Causes Outcome-Level Mode Collapse in Reinforcement Learning and How to Fix It with Inverse Probability Scaling29 Jan

Other references · 19

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  5. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  6. 2021Flow Network based Generative Models for Non-Iterative Diverse Candidate Generationcited by 392
  7. 2021GFlowNet Foundationscited by 290
  8. 2017Molecular De Novo Design through Deep Reinforcement Learningcited by 267
  9. 2022Trajectory balance: Improved credit assignment in GFlowNetscited by 251
  10. 2023Generative Flow Networks as Entropy-Regularized RLcited by 64
  11. 2022Generative Augmented Flow Networkscited by 50
  12. 2023Sample-efficient Multi-objective Molecular Optimization with GFlowNetscited by 50
7 more
  1. 2024Discrete Probabilistic Inference as Control in Multi-path Environmentscited by 44
  2. 2024RGFN: Synthesizable Molecular Generation Using GFlowNetscited by 41
  3. 2023PhyloGFN: Phylogenetic inference with generative flow networkscited by 22
  4. 2026SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoningcited by 11
  5. 2025Symmetry-Aware GFlowNetscited by 7
  6. 2024GFlowNet Training by Policy Gradientscited by 5
  7. 2026Proximal Policy Optimization for Amortized Discrete Samplingcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforcement Learning with Decomposed Subtasks
  2. 2026Beyond Mode Collapse: Distribution Matching for Diverse Reasoning
  3. 2023FP-IRL: Fokker--Planck Inverse Reinforcement Learning -- A Physics-Constrained Approach to Markov Decision Processes
  4. 2026GD2PO: Mitigating Multi-Reward Conflicts via Group-Dynamic reward-Decoupled Policy Optimization
  5. 2026Nonparametric Bayesian Inverse Reinforcement Learning with Data-Parallel Gibbs Sampling
  6. 2026Fast Rates for Inverse Reinforcement Learning
  7. 2026F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare
  8. 2026Inverse Reinforcement Learning without an Optimal Demonstrator: A Feasible Reward Set Approach