mlsanityData to 14 Oct 2026

FIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimization

arXiv 2603.19835 ↗First posted 20 Mar 2026cs.LGcited by 25

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

We present Future-KL Influenced Policy Optimization (FIPO), a reinforcement learning algorithm designed to overcome reasoning bottlenecks in large language models. While GRPO style training scales effectively, it typically relies on outcome-based rewards (ORM) that distribute a global advantage uniformly across every token in a trajectory. We argue that this coarse-grained credit assignment imposes a performance ceiling by failing to distinguish critical logical pivots from trivial tokens. FIPO addresses this by incorporating discounted future-KL divergence into the policy update, creating a dense advantage formulation that re-weights tokens based on their influence on subsequent trajectory behavior. Empirically, FIPO enables models to break through the length stagnation seen in standard baselines. Evaluated on Qwen2.5-32B, FIPO extends the average chain-of-thought length from roughly 4,000 to over 10,000 tokens and increases AIME 2024 Pass@1 accuracy from 50.0% to a peak of 58.0% (converging at approximately 56.0%). This outperforms both DeepSeek-R1-Zero-Math-32B (around 47.0%) and o1-mini (approximately 56.0%). Our results suggest that establishing dense advantage formulations is a vital path for evolving ORM-based algorithms to unlock the full reasoning potential of base models. We open-source our training system, built on the verl framework.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 32 of its 36 references that are in the map. First posted on arXiv 20 Mar 2026.

Builds on

32 of 36 references.

Closest ancestors · 17

Same direction, newest first

  1. 2025Soft Adaptive Policy Optimization25 Nov
  2. 2025Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Model21 Oct
  3. 2025Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination14 Jul
  4. 2025Truncated Proximal Policy Optimization18 Jun
  5. 2025Spurious Rewards: Rethinking Training Signals in RLVR12 Jun
  6. 2025Learning to Reason without External Rewards26 May
  7. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning21 May
  8. 2025TTRL: Test-Time Reinforcement Learning22 Apr
  9. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  10. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  11. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  12. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
5 more
  1. 2025What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret3 Mar
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  3. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  4. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  5. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 15

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  5. 2020Learning to summarize from human feedbackcited by 2,588
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  8. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  9. 2025Group Sequence Policy Optimizationcited by 835
  10. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 467
  11. 2025Skywork Open Reasoner 1 Technical Reportcited by 165
  12. 2019Mastering Complex Control in MOBA Games with Deep Reinforcement Learningcited by 161
3 more
  1. 2024O1 Replication Journey: A Strategic Progress Report -- Part 1cited by 143
  2. 2024O1 Replication Journey -- Part 2: Surpassing O1-preview through Simple Distillation, Big Progress or Bitter Lesson?cited by 92
  3. 2025BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clippingcited by 47

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 25.

25 papers.

Sort:

By area:

  1. 2026VISD: Enhancing Video Reasoning via Structured Self-Distillationcited by 10
  2. 2026DGPO: Distribution Guided Policy Optimization for Fine Grained Credit Assignmentcited by 7
  3. 2026APPO: Agentic Procedural Policy Optimizationcited by 4
  4. 2026VIMPO: Value-Implicit Policy Optimization for LLMscited by 4
  5. 2026Beyond Uniform Token-Level Trust Region in LLM Reinforcement Learningcited by 3
  6. 2026Which Tokens Matter? Adaptive Token Selection for RLVR with the Relative Surprisal Indexcited by 3
  7. 2026One-Way Policy Optimization for Self-Evolving LLMscited by 2
  8. 2026Joint Training of Multi-Token Prediction in Reinforcement Learning via Optimal Coefficient Calibrationcited by 2
  9. 2026PAEC: Position-Aware Entropy Calibration for LLM Reasoning in RLVRcited by 2
  10. 2026Process Advantage Signal Shaping: A Paradigm-Agnostic Middleware for Process-Supervised RL in LLM Reasonerscited by 2
  11. 2026Multi-Step Likelihood-Ratio Correction for Reinforcement Learning with Verifiable Rewardscited by 1
  12. 2026DelTA: Discriminative Token Credit Assignment for Reinforcement Learning from Verifiable Rewardscited by 1
13 more
  1. 2026Clipping Bottleneck: Stabilizing RLVR via Stochastic Recovery of Near-Boundary Signalscited by 1
  2. 2026GMTS: Gradient Magnitude-based Token Selection Improves RLVR Training for LLM Reasoningcited by 1
  3. 2026Not only where, But when: Temporal Scheduling for RLVRcited by 0
  4. 2026ReSum: Synergizing LLM Reasoning and Summarization with Reinforcement Learningcited by 0
  5. 2026ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samplescited by 0
  6. 2026Deconstructing Off-Policy Ratios: Entropy-Normalized Trust Regions for Asynchronous Reinforcement Learningcited by 0
  7. 2026Learning from Hard Prompts: Difficulty-aware Advantage Amplification in Dynamic Samplingcited by 0
  8. 2026Bellman Policy Optimizationcited by 0
  9. 2026Teacher-Student Gaps Are Not Enough: Outcome-Guided On-Policy Distillation for Multi-Turn Autonomous Agentscited by 0
  10. 2026Advancing Entropy-Level Credit Assignment in RLVR via Proximal Entropy Policy Optimizationcited by 0
  11. 2026Semifactual Credit-Augmented Policy Optimizationcited by 0
  12. 2026PEARL: Process-Experience Aware Reinforcement Learning for Multi-Turn Agentscited by 0
  13. 2026LLM Value Functions for Credit Assignment in Agentic RLcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025FAPO: Flawed-Aware Policy Optimization for Efficient and Reliable Reasoning
  2. 2025Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning
  3. 2026Gradient Extrapolation-Based Policy Optimization
  4. 2026AMIR-GRPO: Inducing Implicit Preference Signals into GRPO
  5. 2025DRPO: Efficient Reasoning via Decoupled Reward Policy Optimization
  6. 2025RiskPO: Risk-based Policy Optimization via Verifiable Reward for LLM Post-Training
  7. 2025Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning
  8. 2026Learning from Failures: Correction-Oriented Policy Optimization with Verifiable Rewards