mlsanityData to 14 Oct 2026

When RL Suppresses Its Own Vocabulary: Recovering Reasoning Diversity in Puzzle-to-Math Transfer

arXiv 2605.29190 ↗First posted 28 May 2026cs.LG cs.CLcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning using verifiable rewards (RLVR) improves LLM reasoning, but the conditions under which it transfers across domains -- and why it does so -- remain under-explored. We study cross-domain transfer in a 7B model whose SFT and RL post-training stages use only constraint-satisfaction puzzles, with no mathematics problems in the post-training data. To analyze how transfer emerges, we introduce a reasoning primitive-level framework that combines a 9-class span classifier with motif extraction, allowing us to segment chain-of-thought traces into primitive motifs and track their evolution across training stages and domains. We find that puzzle SFT induces a reasoning-primitive vocabulary, yielding a +7pp textttpass@32 gain on OlymMATH-Hard. Vanilla GSPO then composes these primitives into longer compute-verify chains, adding a further +6pp. However, this RL stage also suppresses exploratory primitives such as textithypothesize and textitbacktrack. To address this, we introduce a novelty bonus that rewards diverse correct rollouts, using perplexity under the reference model as a signal. This restores recovery primitives during RL and adds a further +7pp textttpass@32 relative to vanilla GSPO. Finally, the end-to-end recipe raises the hard-math capability ceiling from 16.0% at the OLMo3-7B-Instruct-SFT base to 36.0%, without adding any mathematics problems during the SFT or RL stages.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 25 of its 37 references that are in the map. First posted on arXiv 28 May 2026.

Builds on

25 of 37 references.

Closest ancestors · 12

Same direction, newest first

  1. 2026Rewarding the Rare: Uniqueness-Aware RL for Creative Problem Solving in LLMs13 Jan
  2. 2025Pass@k Metric for RLVR: A Diagnostic Tool of Exploration, But Not an Objective20 Nov
  3. 2025Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Reward3 Oct
  4. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models14 Aug
  5. 2025LogicPuzzleRL: Cultivating Robust Mathematical Reasoning in LLMs via Reinforcement Learning5 Jun
  6. 2025Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpening3 Jun
  7. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  8. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  9. 2025Evaluating the Diversity and Quality of LLM Generated Content16 Apr
  10. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs3 Mar
  11. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning20 Feb
  12. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan

Other references · 13

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2025Group Sequence Policy Optimizationcited by 835
  4. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 815
  5. 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 297
  6. 2025Olmo 3cited by 129
  7. 2025Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoningcited by 106
  8. 2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problemscited by 87
  9. 2025Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Modelscited by 70
  10. 2025OMEGA: Can LLMs Reason Outside the Box in Math? Evaluating Exploratory, Compositional, and Transformative Generalizationcited by 50
  11. 2025ASTRO: Teaching Language Models to Reason by Reflecting and Backtracking In-Contextcited by 13
  12. 2025Diverse, not Short: A Length-Controlled Data Selection Strategy for Improving Response Diversity of Language Modelscited by 9
1 more
  1. 2025How and Why LLMs Generalize: A Fine-Grained Analysis of LLM Reasoning from Cognitive Behaviors to Low-Level Patternscited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective
  2. 2026How RL Reshapes LLM Reasoning: Transferability, Coverage, and Scaling Laws
  3. 2025Nemotron-CrossThink: Scaling Self-Learning beyond Math Reasoning
  4. 2025How Much Backtracking is Enough? Exploring the Interplay of SFT and RL in Enhancing LLM Reasoning
  5. 2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models
  6. 2025Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection
  7. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains
  8. 2025Can One Domain Help Others? A Data-Centric Study on Multi-Domain Reasoning via Reinforcement Learning