mlsanityData to 14 Oct 2026

When and Why Does Unsupervised RL Succeed in Mathematical Reasoning? A Manifold Envelopment Perspective

arXiv 2603.16578 ↗First posted 17 Mar 2026cs.LG cs.CLcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Although outcome-based reinforcement learning (RL) significantly advances the mathematical reasoning capabilities of Large Language Models (LLMs), its reliance on computationally expensive ground-truth annotations imposes a severe scalability bottleneck. Unsupervised RL guided by intrinsic rewards offers a scalable alternative, yet it suffers from opaque training dynamics and catastrophic instability, such as policy collapse and reward hacking. In this paper, we first design and evaluate a suite of intrinsic rewards that explicitly enforce concise and certain generation. Second, to discover the boundaries of this approach, we test base models across a spectrum of intrinsic reasoning capabilities, revealing how a model's foundational logical prior dictates its success or failure. Finally, to demystify why certain configurations stabilize while others collapse, we introduce a novel geometric diagnostic lens, showing that successful cases are enveloped by manifolds. Ultimately, our work goes beyond merely demonstrating that enforcing concise and certain responses successfully boosts mathematical reasoning; we reveal when this unsupervised approach breaks down and geometrically diagnose why.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 9 of its 10 references that are in the map. First posted on arXiv 17 Mar 2026.

Builds on

9 of 10 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  2. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning21 May
  3. 2023Let's Verify Step by Step31 May
  4. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 5

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  3. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  4. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  5. 2017Soft-DTW: a Differentiable Loss Function for Time-Seriescited by 271

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learning
  2. 2025No Free Lunch: Rethinking Internal Feedback for LLM Reasoning
  3. 2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models
  4. 2026Reward Granularity in RLVR: Comparing Process and Outcome Reward Structures for Mathematical Reasoning in Small Language Models
  5. 2026How RL Reshapes LLM Reasoning: Transferability, Coverage, and Scaling Laws
  6. 2026Beyond Alignment: Expanding Reasoning Capacity via Manifold-Reshaping Policy Optimization
  7. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
  8. 2026Probing the Origins of Reasoning Performance: Representational Quality for Mathematical Problem-Solving in RL vs. SFT Fine-Tuned Models