mlsanityData to 13 Oct 2026

Ignore the KL Penalty! Boosting Exploration on Critical Tokens to Enhance RL Fine-Tuning

arXiv 2502.06533 ↗First posted 10 Feb 2025cs.CL cs.LGcited by 33

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

The ability to achieve long-term goals is a key challenge in the current development of large language models (LLMs). To address this, pre-trained LLMs can be fine-tuned with reinforcement learning (RL) to explore solutions that optimize a given goal. However, exploration with LLMs is difficult, as a balance has to be struck between discovering new solutions and staying close enough to the pre-trained model, so as not to degrade basic capabilities. This is typically controlled with a Kullback-Leibler (KL) penalty. In this paper, we investigate the exploration dynamics of a small language model on a simple arithmetic task. We show how varying degrees of pre-training influence exploration and demonstrate the importance of "critical tokens" which have a dramatic impact on the final outcome. Consequently, we introduce a simple modification to the KL penalty that favors exploration on critical tokens, increasing the efficiency of the RL fine-tuning stage.

Key facts. Cited by 33 papers in the map (data through 13 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 21 of its 36 references that are in the map. First posted on arXiv 10 Feb 2025.

Builds on

21 of 36 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Teaching Large Language Models to Reason with Reinforcement Learning7 Mar

Other references · 20

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,916
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,205
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,270
  5. 2017Deep reinforcement learning from human preferencescited by 3,871
  6. 2020Learning to summarize from human feedbackcited by 2,575
  7. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,223
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,139
  9. 2022Competition-Level Code Generation with AlphaCodecited by 1,445
  10. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 322
  11. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 295
  12. 2019Do NLP Models Know Numbers? Probing Numeracy in Embeddingscited by 158
8 more
  1. 2023How well do Large Language Models perform in Arithmetic tasks?cited by 130
  2. 2020Keep CALM and Explore: Language Models for Action Generation in Text-based Gamescited by 108
  3. 2023Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Taskscited by 84
  4. 2023Positional Description Matters for Transformers Arithmeticcited by 56
  5. 2024Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewardscited by 31
  6. 2024Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generationcited by 22
  7. 2023Chain-of-Thought Reasoning is a Policy Improvement Operatorcited by 8
  8. 2023Conditions for Length Generalization in Learning Reasoning Skillscited by 7

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 18; 2026: 15.

33 papers.

Sort:

By area:

  1. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 548
  2. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 383
  3. 2025Agentic Reinforced Policy Optimizationcited by 168
  4. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 131
  5. 2025DeepDive: Advancing Deep Search Agents with Knowledge Graphs and Multi-Turn RLcited by 54
  6. 2025Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for RLVRcited by 39
  7. 2026Embarrassingly Simple Self-Distillation Improves Code Generationcited by 39
  8. 2025Spotlight on Token Perception for Multimodal Reinforcement Learningcited by 38
  9. 2025VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Modelscited by 36
  10. 2026On the Position Bias of On-Policy Distillationcited by 26
  11. 2025Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimizationcited by 23
  12. 2025Don't Just Fine-tune the Agent, Tune the Environmentcited by 18
21 more
  1. 2025A Comedy of Estimators: On KL Regularization in RL Training of LLMscited by 18
  2. 2025Group Critical-token Policy Optimization for Autoregressive Image Generationcited by 16
  3. 2025SpatialThinker: Reinforcing Scene Graph-Grounded Spatial Reasoning via Dense Rewardscited by 14
  4. 2025Entropy-guided sequence weighting for efficient exploration in RL-based LLM fine-tuningcited by 13
  5. 2026Entropy-Preserving Reinforcement Learningcited by 11
  6. 2026Resolving Action Bottleneck: Agentic Reinforcement Learning Informed by Token-Level Energycited by 5
  7. 2026ParaVT: Taming the Tool Prior Paradox for Parallel Tool Use in Agentic Video Reinforcement Learningcited by 5
  8. 2026PRPO: Perception-Reinforced Policy Optimization via Token-Level Dynamic Advantage Reshapingcited by 5
  9. 2025Self-Reflective Generation at Test Timecited by 3
  10. 2026Training-Trajectory-Aware Token Selectioncited by 3
  11. 2026How Does Reasoning Flow? Tracing Attention-Induced Information Flow for Targeted RL in LLMscited by 3
  12. 2025CogniSQL-R1-Zero: Lightweight Reinforced Reasoning for Efficient SQL Generationcited by 2
  13. 2025ScRPO: From Errors to Insightscited by 1
  14. 2026HTPO: Towards Exploration-Exploitation Balanced Policy Optimization via Hierarchical Token-level Objective Controlcited by 1
  15. 2026Selective Off-Policy Reference Tuning with Plan Guidancecited by 1
  16. 2026Hallucination as Commitment Failure: Larger LLMs Misfire Despite Knowing the Answercited by 1
  17. 2025VCORE: Variance-Controlled Optimization-based Reweighting for Chain-of-Thought Supervisioncited by 0
  18. 2026MHPO: Modulated Hazard-aware Policy Optimization for Stable Reinforcement Learningcited by 0
  19. 2026AtManRL: Towards Faithful Reasoning via Differentiable Attention Saliencycited by 0
  20. 2026ClassicLogic: A Knowledge-Driven Benchmark of Classic Puzzle Games for Evaluating Compositional Generalizationcited by 0
  21. 2026Toward Plasticity-Preserving KL Regularization for Capability Retention in LLM Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
  2. 2025Diversity or Precision? A Deep Dive into Next Token Prediction
  3. 2025Representation-Based Exploration for Language Models: From Test-Time to Post-Training
  4. 2026Entropy-KL Divergence-based Token Masking: A Novel Approach for Selective Fine-tuning of Large Language Models
  5. 2026Offline Exploration-Aware Fine-Tuning for Long-Chain Mathematical Reasoning
  6. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs
  7. 2026Instruction-Conditioned Exploration for Reinforcement Learning with Self-Distillation to an Unconditioned Policy
  8. 2025Recall-Extend Dynamics: Enhancing Small Language Models through Controlled Exploration and Refined Offline Integration