mlsanityData to 7 Oct 2026

Multi-Step Likelihood-Ratio Correction for Reinforcement Learning with Verifiable Rewards

arXiv 2605.20865 ↗First posted 20 May 2026cs.LG cs.AIcited by 1

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning with verifiable rewards (RLVR) plays a pivotal role in improving the reasoning ability of large language models. However, widely used PPO surrogate objectives are fundamentally local, as they rely on a local approximation of the exact policy gradient objective. While this approximation improves stability by reducing the variance induced by importance sampling, it also introduces structural bias into the surrogate objective, which must be controlled through trust region mechanisms. In this work, we introduce the N-step forward trace, which augments the PPO surrogate objective using the cumulative likelihood ratio of the next N-1 tokens. Building on this idea, we propose N-Step Forward-Trace Policy Optimization (NFPO), a practical RLVR algorithm that integrates the N-step forward trace into the masked policy gradient framework. NFPO provides a continuous bridge between the PPO surrogate objective and the exact policy gradient objective, offering a principled mechanism for controlling the bias-variance trade-off. Our theoretical analysis shows that, with an appropriate choice of N, the proposed objective yields a tighter policy-improvement bound than the standard PPO surrogate. Experiments on comprehensive reasoning benchmarks demonstrate that NFPO consistently improves performance, supporting our theoretical findings.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 34 of its 39 references that are in the map. First posted on arXiv 20 May 2026.

Builds on

34 of 39 references.

Closest ancestors · 20

Same direction, newest first

  1. 2026FIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimization20 Mar
  2. 2026Entropy-Preserving Reinforcement Learning12 Mar
  3. 2026Rethinking the Trust Region in LLM Reinforcement Learning4 Feb
  4. 2025Soft Adaptive Policy Optimization25 Nov
  5. 2025Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Model21 Oct
  6. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models14 Aug
  7. 2025Geometric-Mean Policy Optimization28 Jul
  8. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  9. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  10. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning2 Jun
  11. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  12. 2025SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM19 Apr
8 more
  1. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce15 Apr
  2. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning3 Apr
  3. 2025CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models28 Mar
  4. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  6. 2023Let's Verify Step by Step31 May
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  8. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 14

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  7. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  8. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  9. 2025Group Sequence Policy Optimizationcited by 788
  10. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  11. 2025Defeating the Training-Inference Mismatch via FP16cited by 62
  12. 2025Stabilizing Reinforcement Learning with LLMs: Formulation and Practicescited by 38
2 more
  1. 2026Maximum Likelihood Reinforcement Learningcited by 29
  2. 2026Online Causal Kalman Filtering for Stable and Effective Policy Optimizationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Beyond Uniform Token-Level Trust Region in LLM Reinforcement Learningcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning from Failures: Correction-Oriented Policy Optimization with Verifiable Rewards
  2. 2025Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers
  3. 2026Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective
  4. 2026Discounted Beta-Bernoulli Reward Estimation for Sample-Efficient Reinforcement Learning with Verifiable Rewards
  5. 2026What are Key Factors for Updates in RL for LLM Reasoning?
  6. 2026Beyond Uniform Credit Assignment: Selective Eligibility Traces for RLVR
  7. 2025On the optimization dynamics of RLVR: Gradient gap and step size thresholds
  8. 2026LFPO: Likelihood-Free Policy Optimization for Masked Diffusion Models