mlsanityData to 14 Oct 2026

LoRA Scaffolded Policy Optimization (LSPO): A Sampling-Time Low-Rank Scaffold for Recovering Reinforcement-Learning Gradient on Zero-Reward Cliff Prompts

arXiv 2607.27787 ↗First posted 30 Jul 2026cs.LG cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning from verifiable rewards (RLVR) for mathematical reasoning suffers from a structural blind spot: on "cliff" prompts-those on which every sampled rollout in a group fails-the group-normalized advantage is identically zero, so GRPO produces no gradient on precisely the prompts at the frontier of the model's capability. We introduce LoRA Scaffolded Policy Optimization (LSPO), a sampling-time mechanism that recovers this lost gradient. Each RL step, LSPO detects cliff prompts, fits a small low-rank (LoRA) adapter by a brief supervised step on their ground-truth solutions, re-rolls the cliffs with the base-plus-adapter model, splices the now-successful completions back into the RL batch with an importance-sampling correction, and takes a GRPO step on the base alone; the adapter receives only the supervised gradient and is discarded at checkpoint, yielding a base-only model. On DeepMath-103K with DeepSeek-R1-Distill-Qwen-1.5B, evaluated over n=5 paired seeds per arm at a matched 1000-step reporting horizon, LSPO's 5-seed mean matches or beats a DAPO baseline on all 16 (benchmark, pass@k) cells (15 strict wins and one exact tie), with gains of up to +10.7 points on AIME24/pass@4, +6.7 points on AIME24 and AIME26 at pass@16, and +2.4 points on MATH500/pass@1; averaged over the 16 cells the improvement is +3.8 points.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 26 of its 28 references that are in the map. First posted on arXiv 30 Jul 2026.

Builds on

26 of 28 references.

Closest ancestors · 17

Same direction, newest first

  1. 2026RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoning31 May
  2. 2026Learning to Hint for Reinforcement Learning1 Apr
  3. 2026HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation25 Mar
  4. 2026Self-Hinting Language Models Enhance Reinforcement Learning3 Feb
  5. 2025Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning22 Oct
  6. 2025Don't Waste Mistakes: Leveraging Negative RL-Groups via Confidence Reweighting9 Oct
  7. 2025Nudging the Boundaries of LLM Reasoning30 Sep
  8. 2025No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shaping26 Sep
  9. 2025Beyond Two-Stage Training: Cooperative SFT and RL for LLM Reasoning8 Sep
  10. 2025BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoning20 Jun
  11. 2025Adaptive Guidance Accelerates Reinforcement Learning of Reasoning Models16 Jun
  12. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions9 Jun
5 more
  1. 2025Learning to Reason under Off-Policy Guidance21 Apr
  2. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning15 Apr
  3. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  4. 2023Let's Verify Step by Step31 May
  5. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 9

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  6. 2025GHPO: Adaptive Guidance for Stable and Efficient LLM Reinforcement Learningcited by 34
  7. 2025Tina: Tiny Reasoning Models via LoRAcited by 25
  8. 2026Zone of Proximal Policy Optimization: Teacher in Prompts, Not Gradientscited by 3
  9. 2026LatentRevise: Learning from Zero-Hit Reasoningcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026GOPO: Policy Optimization using Ranked Rewards
  2. 2026Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective
  3. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce
  4. 2026Group Adaptive Clipping Policy Optimization
  5. 2026Asymmetric Prompt Weighting for Reinforcement Learning with Verifiable Rewards
  6. 2026ReSPO: Reshaped Sequence Policy Optimization for Gradient Starvation in Off-Policy Learning
  7. 2026How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning
  8. 2026Verifier-Induced Support Reshaping in On-Policy Optimization