Klear-Reasoner: Advancing Reasoning Capability via Gradient-Preserving Clipping Policy Optimization
On the map: LLM reasoning → Reinforcement learning for reasoning
Abstract (arXiv)
We present Klear-Reasoner, a model with long reasoning capabilities that demonstrates careful deliberation during problem solving, achieving outstanding performance across multiple benchmarks. Although there are already many excellent works related to inference models in the current community, there are still many problems with reproducing high-performance inference models due to incomplete disclosure of training details. This report provides an in-depth analysis of the reasoning model, covering the entire post-training workflow from data preparation and long Chain-of-Thought supervised fine-tuning (long CoT SFT) to reinforcement learning (RL), along with detailed ablation studies for each experimental component. For SFT data, our experiments show that a small number of high-quality data sources are more effective than a large number of diverse data sources, and that difficult samples can achieve better results without accuracy filtering. In addition, we investigate two key issues with current clipping mechanisms in RL: Clipping suppresses critical exploration signals and ignores suboptimal trajectories. To address these challenges, we propose Gradient-Preserving clipping Policy Optimization (GPPO) that gently backpropagates gradients from clipped tokens. GPPO not only enhances the model's exploration capacity but also improves its efficiency in learning from negative samples. Klear-Reasoner exhibits exceptional reasoning abilities in mathematics and programming, scoring 90.5% on AIME 2024, 83.2% on AIME 2025, 66.0% on LiveCodeBench V5 and 58.1% on LiveCodeBench V6.
Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 20 of its 28 references that are in the map. First posted on arXiv 11 Aug 2025.
Builds on
20 of 28 references.
Closest ancestors · 9
Same direction, newest first
- 2025Ring-lite: Scalable Reasoning via C3PO-Stabilized Reinforcement Learning for LLMs17 Jun
- 2025AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy16 Jun
- 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning30 May
- 2025AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning22 May
- 2025MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttraining12 May
- 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
- 2017Proximal Policy Optimization Algorithms20 Jul
Other references · 11
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2025Qwen3 Technical Reportcited by 8,820
- 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
- 2024OpenAI o1 System Cardcited by 1,244
- 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 304
- 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 263
- 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
- 2025Skywork Open Reasoner 1 Technical Reportcited by 165
- 2023TACO: Topics in Algorithmic COde generation datasetcited by 160
- 2025OpenCodeReasoning: Advancing Data Distillation for Competitive Codingcited by 113
- 2025OpenCodeReasoning-II: A Simple Test Time Scaling Approach via Self-Critiquecited by 25
Built on by
Citing papers per year: 2025: 11; 2026: 28.
39 papers.
Sort:
By area:
- 2025The Path Not Taken: RLVR Provably Learns Off the Principalscited by 56
- 2025Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?cited by 53
- 2025Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Modelscited by 48
- 2025Agentic Entropy-Balanced Policy Optimizationcited by 40
- 2025Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Rewardcited by 25
- 2025CE-GPPO: Coordinating Entropy via Gradient-Preserving Clipping Policy Optimization in Reinforcement Learningcited by 23
- 2025When Importance Sampling Misallocates Credit: Asymmetric Ratios for Outcome-Supervised RLcited by 21
- 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
- 2025Repurposing Synthetic Data for Fine-grained Search Agent Supervisioncited by 11
- 2026VESPO: Variational Sequence-Level Soft Policy Optimization for Stable Off-Policy LLM Trainingcited by 11
- 2026MASPO: Unifying Gradient Utilization, Probability Mass, and Signal Reliability for Robust and Sample-Efficient LLM Reasoningcited by 11
- 2026LLMs Can Learn to Reason Via Off-Policy RLcited by 10
27 more
- 2026AT2PO: Agentic Turn-based Policy Optimization via Tree Searchcited by 8
- 2026Rethinking Muon Beyond Pretraining: Spectral Failures and High-Pass Remedies for VLA and RLVRcited by 8
- 2025ScaleDiff: Scaling Difficult Problems for Advanced Mathematical Reasoningcited by 6
- 2026Ratio-Variance Regularized Policy Optimization for Efficient LLM Fine-tuningcited by 6
- 2026RAPO: Expanding Exploration for LLM Agents via Retrieval-Augmented Policy Optimizationcited by 6
- 2026From log π to π: Taming Divergence in Soft Clipping via Bilateral Decoupled Decay of Probability Gradient Weightcited by 6
- 2026Orchestrating Tokens and Sequences: Dynamic Hybrid Policy Optimization for RLVRcited by 5
- 2026Missing Old Logits in Asynchronous Agentic RL: Semantic Mismatch and Repair Methods for Off-Policy Correctioncited by 5
- 2026Logics-STEM: Empowering LLM Reasoning via Failure-Driven Post-Training and Document Knowledge Enhancementcited by 4
- 2026APPO: Agentic Procedural Policy Optimizationcited by 4
- 2026QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuningcited by 3
- 2026Beyond Euclidean Clipping: Overcoming Exploration Collapse in LLM RL via Riemannian Isometric Policy Optimizationcited by 3
- 2025Entropy Ratio Clipping as a Soft Global Constraint for Stable Reinforcement Learningcited by 2
- 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
- 2026X-Coder: Advancing Competitive Programming with Synthetic Tasks, Solutions, and Testscited by 1
- 2026D-VLA: A High-Concurrency Distributed Asynchronous Reinforcement Learning Framework for Vision-Language-Action Modelscited by 1
- 2026RLVR Datasets and Where to Find Them: Tracing Data Lineage for Better Training Datacited by 1
- 2026BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoningcited by 1
- 2026ACRL: Adaptive Control of Training-Inference Discrepancy for Stable Reinforcement Learningcited by 1
- 2026Primal Generation, Dual Judgment: Self-Training from Test-Time Scalingcited by 0
- 2026Knowledge-to-Verification: Exploring RLVR for LLMs in Knowledge-Intensive Domainscited by 0
- 2026Ratio-Variance Regularized Policy Optimizationcited by 0
- 2026What are Key Factors for Updates in RL for LLM Reasoning?cited by 0
- 2026UP: Unbounded Positive Asymmetric Optimization for Breaking the Exploration-Stability Dilemmacited by 0
- 2026Lightning Weave: Improving the Accuracy-Efficiency Frontier of Reasoning Models through Capability Compositioncited by 0
- 2026GRRR: The Geometry of Reshaping, Rotation, and Routing in Decoder LLM post-trainingcited by 0
- 2026CATCH: A Controllable Analysis Testbed for Reward Hacking in Coding RLcited by 0
- 2026CATCH: A Controllable Analysis Testbed for Reward Hacking in Coding RL30 Sep 2026
- 2026Lightning Weave: Improving the Accuracy-Efficiency Frontier of Reasoning Models through Capability Composition13 Sep 2026
- 2026GRRR: The Geometry of Reshaping, Rotation, and Routing in Decoder LLM post-training26 Aug 2026
- 2026ACRL: Adaptive Control of Training-Inference Discrepancy for Stable Reinforcement Learning27 Jul 2026
- 2026Beyond Euclidean Clipping: Overcoming Exploration Collapse in LLM RL via Riemannian Isometric Policy Optimization11 Jul 2026
- 2026UP: Unbounded Positive Asymmetric Optimization for Breaking the Exploration-Stability Dilemma8 Jul 2026
- 2026What are Key Factors for Updates in RL for LLM Reasoning?21 Jun 2026
- 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learning20 Jun 2026
- 2026APPO: Agentic Procedural Policy Optimization10 Jun 2026
- 2026BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoning26 May 2026
- 2026RLVR Datasets and Where to Find Them: Tracing Data Lineage for Better Training Data26 May 2026
- 2026Ratio-Variance Regularized Policy Optimization26 May 2026
27 more
- 2026Rethinking Muon Beyond Pretraining: Spectral Failures and High-Pass Remedies for VLA and RLVR19 May 2026
- 2026Knowledge-to-Verification: Exploring RLVR for LLMs in Knowledge-Intensive Domains18 May 2026
- 2026D-VLA: A High-Concurrency Distributed Asynchronous Reinforcement Learning Framework for Vision-Language-Action Models13 May 2026
- 2026Missing Old Logits in Asynchronous Agentic RL: Semantic Mismatch and Repair Methods for Off-Policy Correction12 May 2026
- 2026Primal Generation, Dual Judgment: Self-Training from Test-Time Scaling11 May 2026
- 2026From log π to π: Taming Divergence in Soft Clipping via Bilateral Decoupled Decay of Probability Gradient Weight15 Mar 2026
- 2026RAPO: Expanding Exploration for LLM Agents via Retrieval-Augmented Policy Optimization3 Mar 2026
- 2026LLMs Can Learn to Reason Via Off-Policy RL22 Feb 2026
- 2026MASPO: Unifying Gradient Utilization, Probability Mass, and Signal Reliability for Robust and Sample-Efficient LLM Reasoning19 Feb 2026
- 2026VESPO: Variational Sequence-Level Soft Policy Optimization for Stable Off-Policy LLM Training11 Feb 2026
- 2026QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuning4 Feb 2026
- 2026X-Coder: Advancing Competitive Programming with Synthetic Tasks, Solutions, and Tests11 Jan 2026
- 2026Orchestrating Tokens and Sequences: Dynamic Hybrid Policy Optimization for RLVR9 Jan 2026
- 2026AT2PO: Agentic Turn-based Policy Optimization via Tree Search8 Jan 2026
- 2026Ratio-Variance Regularized Policy Optimization for Efficient LLM Fine-tuning6 Jan 2026
- 2026Logics-STEM: Empowering LLM Reasoning via Failure-Driven Post-Training and Document Knowledge Enhancement4 Jan 2026
- 2025Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Models15 Dec 2025
- 2025Entropy Ratio Clipping as a Soft Global Constraint for Stable Reinforcement Learning5 Dec 2025
- 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence23 Nov 2025
- 2025The Path Not Taken: RLVR Provably Learns Off the Principals11 Nov 2025
- 2025Repurposing Synthetic Data for Fine-grained Search Agent Supervision28 Oct 2025
- 2025Agentic Entropy-Balanced Policy Optimization16 Oct 2025
- 2025When Importance Sampling Misallocates Credit: Asymmetric Ratios for Outcome-Supervised RL7 Oct 2025
- 2025Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Reward3 Oct 2025
- 2025Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?1 Oct 2025
- 2025ScaleDiff: Scaling Difficult Problems for Advanced Mathematical Reasoning25 Sep 2025
- 2025CE-GPPO: Coordinating Entropy via Gradient-Preserving Clipping Policy Optimization in Reinforcement Learning25 Sep 2025
- 2025The Path Not Taken: RLVR Provably Learns Off the Principalscited by 56
- 2025Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?cited by 53
- 2025Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Modelscited by 48
- 2025Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Rewardcited by 25
- 2025CE-GPPO: Coordinating Entropy via Gradient-Preserving Clipping Policy Optimization in Reinforcement Learningcited by 23
- 2025Repurposing Synthetic Data for Fine-grained Search Agent Supervisioncited by 11
- 2026MASPO: Unifying Gradient Utilization, Probability Mass, and Signal Reliability for Robust and Sample-Efficient LLM Reasoningcited by 11
- 2026LLMs Can Learn to Reason Via Off-Policy RLcited by 10
- 2026Rethinking Muon Beyond Pretraining: Spectral Failures and High-Pass Remedies for VLA and RLVRcited by 8
- 2025ScaleDiff: Scaling Difficult Problems for Advanced Mathematical Reasoningcited by 6
- 2026Ratio-Variance Regularized Policy Optimization for Efficient LLM Fine-tuningcited by 6
- 2026From log π to π: Taming Divergence in Soft Clipping via Bilateral Decoupled Decay of Probability Gradient Weightcited by 6
12 most cited of 23 All 23 (JSON)
- 2026Lightning Weave: Improving the Accuracy-Efficiency Frontier of Reasoning Models through Capability Composition13 Sep 2026
- 2026Beyond Euclidean Clipping: Overcoming Exploration Collapse in LLM RL via Riemannian Isometric Policy Optimization11 Jul 2026
- 2026UP: Unbounded Positive Asymmetric Optimization for Breaking the Exploration-Stability Dilemma8 Jul 2026
- 2026What are Key Factors for Updates in RL for LLM Reasoning?21 Jun 2026
- 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learning20 Jun 2026
- 2026BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoning26 May 2026
- 2026Rethinking Muon Beyond Pretraining: Spectral Failures and High-Pass Remedies for VLA and RLVR19 May 2026
- 2026Knowledge-to-Verification: Exploring RLVR for LLMs in Knowledge-Intensive Domains18 May 2026
- 2026Primal Generation, Dual Judgment: Self-Training from Test-Time Scaling11 May 2026
- 2026From log π to π: Taming Divergence in Soft Clipping via Bilateral Decoupled Decay of Probability Gradient Weight15 Mar 2026
- 2026LLMs Can Learn to Reason Via Off-Policy RL22 Feb 2026
- 2026MASPO: Unifying Gradient Utilization, Probability Mass, and Signal Reliability for Robust and Sample-Efficient LLM Reasoning19 Feb 2026
12 newest of 23 All 23 (JSON)
- 2025Agentic Entropy-Balanced Policy Optimizationcited by 40
- 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
- 2026AT2PO: Agentic Turn-based Policy Optimization via Tree Searchcited by 8
- 2026Missing Old Logits in Asynchronous Agentic RL: Semantic Mismatch and Repair Methods for Off-Policy Correctioncited by 5
- 2026APPO: Agentic Procedural Policy Optimizationcited by 4
- 2026APPO: Agentic Procedural Policy Optimization10 Jun 2026
- 2026Missing Old Logits in Asynchronous Agentic RL: Semantic Mismatch and Repair Methods for Off-Policy Correction12 May 2026
- 2026AT2PO: Agentic Turn-based Policy Optimization via Tree Search8 Jan 2026
- 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence23 Nov 2025
- 2025Agentic Entropy-Balanced Policy Optimization16 Oct 2025
- 2025When Importance Sampling Misallocates Credit: Asymmetric Ratios for Outcome-Supervised RLcited by 21
- 2026X-Coder: Advancing Competitive Programming with Synthetic Tasks, Solutions, and Testscited by 1
- 2026RLVR Datasets and Where to Find Them: Tracing Data Lineage for Better Training Datacited by 1
- 2026Ratio-Variance Regularized Policy Optimizationcited by 0
- 2026GRRR: The Geometry of Reshaping, Rotation, and Routing in Decoder LLM post-trainingcited by 0
- 2026GRRR: The Geometry of Reshaping, Rotation, and Routing in Decoder LLM post-training26 Aug 2026
- 2026RLVR Datasets and Where to Find Them: Tracing Data Lineage for Better Training Data26 May 2026
- 2026Ratio-Variance Regularized Policy Optimization26 May 2026
- 2026X-Coder: Advancing Competitive Programming with Synthetic Tasks, Solutions, and Tests11 Jan 2026
- 2025When Importance Sampling Misallocates Credit: Asymmetric Ratios for Outcome-Supervised RL7 Oct 2025
Similar, not cited
Close in wording, no citation either way
- 2026On-Policy Supervised Fine-Tuning for Efficient Reasoning
- 2025SmartThinker: Learning to Compress and Preserve Reasoning by Step-Level Length Control
- 2024RL-STaR: Theoretical Analysis of Reinforcement Learning Frameworks for Self-Taught Reasoner
- 2026ConMax: Confidence-Maximizing Compression for Efficient Chain-of-Thought Reasoning
- 2025Reasoning Vectors: Transferring Chain-of-Thought Capabilities via Task Arithmetic
- 2025ConciseRL: Conciseness-Guided Reinforcement Learning for Efficient Reasoning Models
- 2026Drop the Act: Probe-Filtered RL for Faithful Chain-of-Thought Reasoning
- 2025How Far Are We from Optimal Reasoning Efficiency?