mlsanityData to 7 Oct 2026

SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoning

arXiv 2602.01062 ↗First posted 1 Feb 2026cs.AIcited by 9

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning with verifiable rewards has shown notable effectiveness in enhancing large language models (LLMs) reasoning performance, especially in mathematics tasks. However, such improvements often come with reduced outcome diversity, where the model concentrates probability mass on a narrow set of solutions. Motivated by diminishing-returns principles, we introduce a set level diversity objective defined over sampled trajectories using kernelized similarity. Our approach derives a leave-one-out marginal contribution for each sampled trajectory and integrates this objective as a plug-in advantage shaping term for policy optimization. We further investigate the contribution of a single trajectory to language model diversity within a distribution perturbation framework. This analysis theoretically confirms a monotonicity property, proving that rarer trajectories yield consistently higher marginal contributions to the global diversity. Extensive experiments across a range of model scales demonstrate the effectiveness of our proposed algorithm, consistently outperforming strong baselines in both Pass@1 and Pass@K across various benchmarks.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 36 of its 38 references that are in the map. First posted on arXiv 1 Feb 2026.

Builds on

36 of 38 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models14 Aug
  2. 2025Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning13 Aug
  3. 2025Geometric-Mean Policy Optimization28 Jul
  4. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  5. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  6. 2025Diversity-Aware Policy Optimization for Large Language Model Reasoning29 May
  7. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  8. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization18 May
  9. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  10. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning3 Apr
  11. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  12. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
4 more
  1. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  2. 2025Diverse Preference Optimization30 Jan
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  4. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 20

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2024Qwen2 Technical Reportcited by 2,076
  7. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  8. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  9. 2025Group Sequence Policy Optimizationcited by 788
  10. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
  11. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 158
  12. 2024MathScale: Scaling Instruction Tuning for Mathematical Reasoningcited by 144
8 more
  1. 2024Language Model Cascades: Token-level uncertainty and beyondcited by 115
  2. 2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problemscited by 83
  3. 2025Jointly Reinforcing Diversity and Quality in Language Model Generationscited by 79
  4. 2023Language Model Self-improvement by Reinforcement Learning Contemplationcited by 75
  5. 2025Optimizing Language Models for Inference Time Objectives using Reinforcement Learningcited by 49
  6. 2025Modifying Large Language Model Post-Training for Diverse Creative Writingcited by 39
  7. 2019Diversity-Inducing Policy Gradient: Using Maximum Mean Discrepancy to Find a Set of Diverse Policiescited by 34
  8. 2025Creative Preference Optimizationcited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 9.

9 papers.

  1. 2026Emergence of Exploration in Policy Gradient Reinforcement Learning via Retryingcited by 5
  2. 2026Leveraging Error Diversity in Group Rollouts for Reinforcement Learningcited by 2
  3. 2026Multi-Axis Max@K Reinforcement Learning for Representative Diversity in Text-to-Image Generationcited by 2
  4. 2026When Correct Solutions Repeat: Rarity-Aware Credit Redistribution for GRPOcited by 0
  5. 2026Strategically Diverse Sampling for Self-Trainingcited by 0
  6. 2026Improving the Diversity of LLM Outputs without a Trade-offcited by 0
  7. 2026SERA: Scale-Equalized Rollout Allocation for Maximum Likelihood Reinforcement Learningcited by 0
  8. 2026Measuring and Mitigating Solution Mode Collapse in RLVRcited by 0
  9. 2026Beyond Action Entropy: Quotient-Space Exploration for Genome-Scale Metabolic Model Repaircited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Outcome-based Exploration for LLM Reasoning
  2. 2026Back to Basics: Revisiting Exploration in Reinforcement Learning for LLM Reasoning via Generative Probabilities
  3. 2026DSDR: Dual-Scale Diversity Regularization for Exploration in LLM Reasoning
  4. 2025Whatever Remains Must Be True: Filtering Drives Reasoning in LLMs, Shaping Diversity
  5. 2026Rewarding the Rare: Uniqueness-Aware RL for Creative Problem Solving in LLMs
  6. 2025Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning
  7. 2025Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewards
  8. 2026Diverse Thinking Schemata Elicit Better Reasoning in Large Language Models