mlsanityData to 14 Oct 2026

Skip-Connected Policy Optimization for Implicit Advantage

arXiv 2604.08690 ↗First posted 9 Apr 2026cs.LG cs.CLcited by 1

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Group Relative Policy Optimization (GRPO) has proven effective in RLVR by using outcome-based rewards. While fine-grained dense rewards can theoretically improve performance, we reveal that under practical sampling budgets, Monte Carlo estimation yields high-variance and sign-inconsistent advantages for early reasoning tokens, paradoxically underperforming outcome-only GRPO. We propose Skip-Connected Optimization (SKPO), which decomposes reasoning into upstream and downstream phases: upstream receives dense rewards from downstream Monte Carlo sampling with single-stream optimization; downstream maintains group-relative optimization, where a skip connection concatenates the upstream segment with the original problem, enabling the model to leverage helpful upstream reasoning while preserving the freedom to bypass flawed reasoning through direct problem access. Experiments demonstrate improvements of 3.91% and 6.17% relative gains over the strongest baselines on Qwen2.5-Math-7B and Llama-3.2-3B respectively across mathematical benchmarks and out-of-domain tasks including general reasoning and code generation. Further analysis reveals an implicit advantage: SKPO generates trajectories with higher intermediate-step quality even when matched for final correctness.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 53 of its 61 references that are in the map. First posted on arXiv 9 Apr 2026.

Builds on

53 of 61 references.

Closest ancestors · 30

Same direction, newest first

  1. 2025ESPO: Entropy Importance Sampling Policy Optimization29 Nov
  2. 2025Soft Adaptive Policy Optimization25 Nov
  3. 2025MURPHY: Feedback-Aware GRPO with Retrospective Credit Assignment for Multi-Turn Code Generation11 Nov
  4. 2025ScRPO: From Errors to Insights8 Nov
  5. 2025Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Markov Likelihood10 Oct
  6. 2025DRPO: Efficient Reasoning via Decoupled Reward Policy Optimization6 Oct
  7. 2025What Can You Do When You Have Zero Rewards During RL?4 Oct
  8. 2025It Takes Two: Your GRPO Is Secretly DPO1 Oct
  9. 2025Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friends29 Sep
  10. 2025GRPO is Secretly a Process Reward Model25 Sep
  11. 2025Single-stream Policy Optimization16 Sep
  12. 2025Emergent Hierarchical Reasoning in LLMs through Reinforcement Learning3 Sep
18 more
  1. 2025DCPO: Dynamic Clipping Policy Optimization2 Sep
  2. 2025ReCode: Reinforcing Code Generation with Reasoning-Process Rewards7 Aug
  3. 2025CAPO: Towards Enhancing LLM Reasoning through Generative Credit Assignment4 Aug
  4. 2025Geometric-Mean Policy Optimization28 Jul
  5. 2025Multi-Layer GRPO: Enhancing Reasoning and Self-Correction in Large Language Models5 Jun
  6. 2025Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback3 Jun
  7. 2025Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpening3 Jun
  8. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  9. 2025Discriminative Policy Optimization for Token-Level Reward Models29 May
  10. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression27 May
  11. 2025Self-Evolving Curriculum for LLM Reasoning20 May
  12. 2025Token-Efficient RL for LLM Reasoning29 Apr
  13. 2025TTRL: Test-Time Reinforcement Learning22 Apr
  14. 2025Learning to Reason under Off-Policy Guidance21 Apr
  15. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  16. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  17. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs3 Mar
  18. 2024What Are Step-Level Reward Models Rewarding? Counterintuitive Findings from MCTS-Boosted Mathematical Reasoning20 Dec

Other references · 23

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  4. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  5. 2025Group Sequence Policy Optimizationcited by 835
  6. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
  7. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 263
  8. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 228
  9. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 187
  10. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 183
  11. 2025The Art of Scaling Reinforcement Learning Compute for LLMscited by 101
  12. 2025JustRL: Scaling a 1.5B LLM with a Simple RL Recipecited by 64
11 more
  1. 2025ReasonFlux: Hierarchical LLM Reasoning via Scaling Thought Templatescited by 63
  2. 2025DLER: Doing Length pEnalty Right - Incentivizing More Intelligence per Token via Reinforcement Learningcited by 35
  3. 2025CriticSearch: Fine-Grained Credit Assignment for Search Agents via a Retrospective Criticcited by 30
  4. 2025SPELL: Self-Play Reinforcement Learning for Evolving Long-Context Language Modelscited by 29
  5. 2025PAG: Multi-Turn Reinforced LLM Self-Correction with Policy as Generative Verifiercited by 28
  6. 2025Promoting Efficient Reasoning with Verifiable Stepwise Rewardcited by 28
  7. 2024Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcementcited by 12
  8. 2024RED: Unleashing Token-Level Rewards from Holistic Feedback via Reward Redistributioncited by 10
  9. 2025GPO: Learning from Critical Steps to Improve LLM Reasoningcited by 10
  10. 2025Response-Level Rewards Are All You Need for Online Reinforcement Learning in LLMs: A Mathematical Perspectivecited by 5
  11. 2025GEPO: Group Expectation Policy Optimization for Stable Heterogeneous Reinforcement Learningcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Entropy Polarity in Reinforcement Fine-Tuning: Direction, Asymmetry, and Controlcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2026GRPO-VPS: Enhancing Group Relative Policy Optimization with Verifiable Process Supervision for Effective Reasoning
  2. 2026BPPO: Binary Prefix Policy Optimization for Efficient GRPO-Style Reasoning RL with Concise Responses
  3. 2026Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation
  4. 2026VeriGate: Verifier-Gated Step-Level Supervision for GRPO
  5. 2026MCPO: Mastery-Consolidated Policy Optimization for Large Reasoning Models
  6. 2026Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective
  7. 2025Sharpness-Guided Group Relative Policy Optimization via Probability Shaping
  8. 2026RRPO: Reference-Relative Policy Optimization with Stratified Conditional Rollouts