mlsanityData to 7 Oct 2026

Rethinking Probability-Based Reinforcement Learning From Posterior Concentration

arXiv 2610.01458 ↗First posted 1 Oct 2026cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Verifier-free reinforcement learning with probability-based rewards offers a promising way to train LLMs on general reasoning tasks where external verifiers are unavailable. Yet the reliability of these rewards, especially in long-horizon reasoning, remains underexplored. This work identifies a length-dependent failure mode of probability rewards, which we call the Posterior Concentration Phenomenon (PCP). We show that the probability of a reference answer conditioned on a reasoning trace often collapses to a low-variance interval as the trace becomes lengthy. This phenomenon results in nearly indistinguishable rewards, which, under GRPO-based settings, makes probability-based policy optimization unstable and inefficient. Motivated by this, we propose Reinforcement Learning with Concentration-aware Posterior Rewards (RLCPR), a verifier-free RL framework to explicitly account for PCP for better optimization stability and token efficiency. It has two components: uncertainty-aware data sampling, which reduces concentration-prone rollouts before generation, and concentration-aware regularization, which penalizes unnecessarily long traces when posterior rewards collapse. Extensive experiments show that, alongside higher token efficiency, RLCPR outperforms the state-of-the-art verifier-free RL baseline by up to 4.0% on six of seven benchmarks, including general-domain and mathematical reasoning challenges.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 20 of its 37 references that are in the map. First posted on arXiv 1 Oct 2026.

Builds on

20 of 37 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025RLPR: Extrapolating RLVR to General Domains without Verifiers23 Jun
  2. 2025Reinforcing General Reasoning without Verifiers27 May
  3. 2025General-Reasoner: Advancing LLM Reasoning Across All Domains20 May
  4. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  5. 2025Process Reinforcement through Implicit Rewards3 Feb
  6. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  7. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  8. 2023Let's Verify Step by Step31 May

Other references · 12

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  5. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  6. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  7. 2025Beyond Verifiable Rewards: Scaling Reinforcement Learning for Language Models to Unverifiable Datacited by 29
  8. 2025LLM Probability Concentration: How Alignment Shrinks the Generative Horizoncited by 26
  9. 2026The Cost of Reasoning: Chain-of-Thought Induces Overconfidence in Vision-Language Modelscited by 5
  10. 2024Slaves to the Law of Large Numbers: An Asymptotic Equipartition Property for Perplexity in Generative Language Modelscited by 4
  11. 2026CoRA: Confidence-Rationale Alignment for Reliable Chain-of-Thought Reasoningcited by 1
  12. 2026Aligning Thoughts with Answers: Probability Rewards to Tame Thinking Driftcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Unlocking the Critic: Reward-Free Policy Optimization for LLM Post-Training
  2. 2025Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Reward
  3. 2026Back to Basics: Revisiting Exploration in Reinforcement Learning for LLM Reasoning via Generative Probabilities
  4. 2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models
  5. 2025Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning
  6. 2026Learning to Explore with Parameter-Space Noise: A Deep Dive into Parameter-Space Noise for Reinforcement Learning with Verifiable Rewards
  7. 2026Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement
  8. 2025PACR: Progressively Ascending Confidence Reward for LLM Reasoning