mlsanityData to 7 Oct 2026

Beyond Normalization: Rethinking the Partition Function as a Difficulty Scheduler for RLVR

arXiv 2602.12642 ↗First posted 13 Feb 2026cs.CL cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reward-maximizing RL methods have shown to be capable of enhancing the reasoning performance of LLMs, but often lead to reduced generation diversity. Recent works address this issue by adopting GFlowNets, training LLMs to match a target distribution while jointly learning its partition function. In contrast to prior works that treat this partition function solely as a normalizer, we reinterpret it as a per-prompt expected-reward (i.e., online accuracy) signal, leveraging this unused information to improve sample efficiency. Specifically, we first establish a theoretical relationship between the partition function and per-prompt accuracy estimates. Building on this key insight, we propose Partition Function-Guided RL (PACED-RL), a post-training framework that leverages accuracy estimates to prioritize informative question prompts during training, and further improves sample efficiency through an accuracy estimate error-prioritized replay. Crucially, both components reuse information already produced during GFlowNet training, effectively amortizing the compute overhead into the existing optimization process. Extensive experiments across diverse benchmarks demonstrate strong performance improvements over GRPO and prior GFlowNet approaches, highlighting PACED-RL as a promising direction for a more sample efficient distribution-matching training for LLMs.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 50 of its 51 references that are in the map. First posted on arXiv 13 Feb 2026.

Builds on

50 of 51 references.

Closest ancestors · 18

Same direction, newest first

  1. 2025Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Models15 Dec
  2. 2025CurES: From Gradient Analysis to Efficient Curriculum Learning for Reasoning LLMs1 Oct
  3. 2025FlowRL: Matching Reward Distributions for LLM Reasoning18 Sep
  4. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models14 Aug
  5. 2025Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?7 Jul
  6. 2025EFRame: Deeper Reasoning via Exploration-Filter-Replay Reinforcement Learning Framework27 Jun
  7. 2025RePO: Replay-Enhanced Policy Optimization11 Jun
  8. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts2 Jun
  9. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning2 Jun
  10. 2025Learning Like Humans: Advancing LLM Reasoning Capabilities via Adaptive Difficulty Curriculum Learning and Expert-Guided Self-Reformulation13 May
  11. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  12. 2025Evaluating the Diversity and Quality of LLM Generated Content16 Apr
6 more
  1. 2025Efficient Reinforcement Finetuning via Adaptive Curriculum Learning7 Apr
  2. 2025Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning4 Apr
  3. 2025Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training24 Mar
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  5. 2025Learning to Reason at the Frontier of Learnability17 Feb
  6. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 32

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  7. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  8. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  9. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  10. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  11. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  12. 2017Constrained Policy Optimizationcited by 928
20 more
  1. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  2. 2014New insights and perspectives on the natural gradient methodcited by 435
  3. 2021GFlowNet Foundationscited by 286
  4. 2018Distributed Prioritized Experience Replaycited by 260
  5. 2022Trajectory balance: Improved credit assignment in GFlowNetscited by 244
  6. 2023Does Writing with Language Models Reduce Content Diversity?cited by 190
  7. 2020Revisiting Fundamentals of Experience Replaycited by 171
  8. 2022Learning GFlowNets from partial episodes for improved convergence and stabilitycited by 129
  9. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 102
  10. 2025MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbationscited by 88
  11. 2025Jointly Reinforcing Diversity and Quality in Language Model Generationscited by 79
  12. 2023Towards Understanding and Improving GFlowNet Trainingcited by 74
  13. 2024Amortizing intractable inference in diffusion models for vision, language, and controlcited by 70
  14. 2024Preserving Diversity in Supervised Fine-Tuning of Large Language Modelscited by 68
  15. 2023Robust Scheduling with GFlowNetscited by 67
  16. 2024Learning diverse attacks on large language models for robust red-teaming and safety tuningcited by 54
  17. 2023Local Search GFlowNetscited by 45
  18. 2024Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examplescited by 29
  19. 2022Prioritizing Samples in Reinforcement Learning with Reducible Losscited by 20
  20. 2024GDPO: Learning to Directly Align Language Models with Diversity Using GFlowNetscited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026GFlowRL: Scaling Distribution-Matching RL to Large Language Models
  2. 2025Limits of Generalization in RLVR: Two Case Studies in Mathematical Reasoning
  3. 2026Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMs
  4. 2026On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation
  5. 2026Learning from Less: Measuring the Effectiveness of RLVR in Low Data and Compute Regimes
  6. 2026Decouple before Integration: Test-time Synthesis of SFT and RLVR Task Vectors
  7. 2025Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration
  8. 2025TinyV: Reducing False Negatives in Verification Improves RL for LLM Reasoning