# KnowRL: Boosting LLM Reasoning via Reinforcement Learning with Minimal-Sufficient Knowledge Guidance

arXiv: https://arxiv.org/abs/2604.12627 · HTML: https://mlsanity.com/paper/2604.12627/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 26 of its 35 references that are in the map.
- First posted on arXiv 14 Apr 2026.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 26 of 26)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Kimi K2.5: Visual Agentic Intelligence](https://mlsanity.com/paper/2602.02276/) (2026)
- [AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning dataset](https://mlsanity.com/paper/2504.16891/) (2025)
- [JustRL: Scaling a 1.5B LLM with a Simple RL Recipe](https://mlsanity.com/paper/2512.16649/) (2025)
- [BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoning](https://mlsanity.com/paper/2506.17211/) (2025)
- [DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPO](https://mlsanity.com/paper/2506.07464/) (2025)
- [Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling](https://mlsanity.com/paper/2507.01679/) (2025)
- [UFT: Unifying Supervised and Reinforcement Fine-Tuning](https://mlsanity.com/paper/2505.16984/) (2025)
- [Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning](https://mlsanity.com/paper/2510.19807/) (2025)
- [QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentation](https://mlsanity.com/paper/2507.13266/) (2025)
- [POPE: Learning to Reason on Hard Problems via Privileged On-Policy Exploration](https://mlsanity.com/paper/2601.18779/) (2026)
- [StepHint: Multi-level Stepwise Hints Enhance Reinforcement Learning to Reason](https://mlsanity.com/paper/2507.02841/) (2025)
- [GHPO: Adaptive Guidance for Stable and Efficient LLM Reinforcement Learning](https://mlsanity.com/paper/2507.10628/) (2025)
- [Boosting MLLM Reasoning with Text-Debiased Hint-GRPO](https://mlsanity.com/paper/2503.23905/) (2025)
- [TemplateRL: Structured Template-Guided Reinforcement Learning for LLM Reasoning](https://mlsanity.com/paper/2505.15692/) (2025)
- [RLAD: Training LLMs to Discover Abstractions for Solving Reasoning Problems](https://mlsanity.com/paper/2510.02263/) (2025)
- [Revisiting Entropy in Reinforcement Learning for Large Reasoning Models](https://mlsanity.com/paper/2511.05993/) (2025)
- [Self-Hinting Language Models Enhance Reinforcement Learning](https://mlsanity.com/paper/2602.03143/) (2026)
- [Adaptive Guidance Accelerates Reinforcement Learning of Reasoning Models](https://mlsanity.com/paper/2506.13923/) (2025)
- [Don't Tell the Answer, Truly Guide the Reasoning During RL Rollouts](https://mlsanity.com/paper/2510.09388/) (2025)
- [ADHint: Adaptive Hints with Difficulty Priors for Reinforcement Learning](https://mlsanity.com/paper/2512.13095/) (2025)
- [ERNIE 5.0 Technical Report](https://mlsanity.com/paper/2602.04705/) (2026)
- [G2RPO-A: Guided Group Relative Policy Optimization with Adaptive Guidance](https://mlsanity.com/paper/2508.13023/) (2025)
- [ATTNPO: Attention-Guided Process Supervision for Efficient Reasoning](https://mlsanity.com/paper/2602.09953/) (2026)

## Built on by (most cited first, 8 of 8)

- [Co-Evolving Policy Distillation](https://mlsanity.com/paper/2604.27083/) (2026)
- [Incentivizing Parametric Knowledge via Reinforcement Learning with Verifiable Rewards for Cross-Cultural Entity Translation](https://mlsanity.com/paper/2604.16881/) (2026)
- [DenoiseRL: Bootstrapping Reasoning Models to Recover from Noisy Prefixes](https://mlsanity.com/paper/2605.28421/) (2026)
- [AdaKP: Online Adaptive Knowledge-Point Selection for Reasoning-Oriented Reinforcement Learning](https://mlsanity.com/paper/2607.24833/) (2026)
- [Continual Learning in Transition](https://mlsanity.com/paper/2608.06216/) (2026)
- [Learning from Hard Prompts: Difficulty-aware Advantage Amplification in Dynamic Sampling](https://mlsanity.com/paper/2608.27982/) (2026)
- [Teach to Learn: Hint Annealing for Self-improving LLM Reasoning](https://mlsanity.com/paper/2609.34975/) (2026)
- [Splicing Hints into Reasoning Traces: Learning from Demonstrator Failures](https://mlsanity.com/paper/2610.13996/) (2026)

## Abstract (arXiv)

RLVR improves reasoning in large language models, but its effectiveness is often limited by severe reward sparsity on hard problems. Recent hint-based RL methods mitigate sparsity by injecting partial solutions or abstract templates, yet they typically scale guidance by adding more tokens, which introduce redundancy, inconsistency, and extra training overhead. We propose KnowRL (Knowledge-Guided Reinforcement Learning), an RL training framework that treats hint design as a minimal-sufficient guidance problem. During RL training, KnowRL decomposes guidance into atomic knowledge points (KPs) and uses Constrained Subset Search (CSS) to construct compact, interaction-aware subsets for training. We further identify a pruning interaction paradox -- removing one KP may help while removing multiple such KPs can hurt -- and explicitly optimize for robust subset curation under this dependency structure. We train KnowRL-Nemotron-1.5B from OpenMath-Nemotron-1.5B. Across eight reasoning benchmarks at the 1.5B scale, KnowRL-Nemotron-1.5B consistently outperforms strong RL and hinting baselines. Without KP hints at inference, KnowRL-Nemotron-1.5B reaches 70.08 average accuracy, already surpassing Nemotron-1.5B by +9.63 points; with selected KPs, performance improves to 74.16, establishing a new state of the art at this scale. The model, curated training data, and code are publicly available at https://github.com/Hasuer/KnowRL.