mlsanityData to 14 Oct 2026

LamPO: A Lambda Style Policy Optimization for Reasoning Language Models

arXiv 2605.21235 ↗First posted 20 May 2026cs.CLcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning with verifiable rewards (RLVR) has become an effective paradigm for improving reasoning language models on tasks such as mathematics, coding, and scientific question answering. However, widely used group-relative objectives, such as GRPO, summarize each sampled group with scalar statistics and therefore discard fine-grained relational information among candidate responses. This weakens credit assignment under sparse outcome rewards, especially when multiple generated solutions differ only subtly in reasoning quality. We propose LamPO, a Lambda-Style Policy Optimization method that replaces scalar group advantages with a Pairwise Decomposed Advantage. LamPO aggregates pairwise reward gaps within each response group and modulates each comparison by a confidence-aware weight computed from sequence log-probability differences, while retaining the critic-free and clipped-update structure of PPO-style optimization. When reference solutions are available, we further add a lightweight ROUGE-L-based dense auxiliary reward to reduce reward sparsity. Experiments on AIME24, AIME25, MATH-500, and GPQA-Diamond with Qwen3-1.7B, Qwen3-4B, and Phi-4-mini show that LamPO consistently improves over GRPO and recent RLVR variants, with more stable training dynamics and better sample efficiency.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 16 of its 19 references that are in the map. First posted on arXiv 20 May 2026.

Builds on

16 of 19 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Soft Adaptive Policy Optimization25 Nov
  2. 2025RLPR: Extrapolating RLVR to General Domains without Verifiers23 Jun
  3. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  4. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  6. 2025Process Reinforcement through Implicit Rewards3 Feb
  7. 2023Let's Verify Step by Step31 May
  8. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 8

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  5. 2025Group Sequence Policy Optimizationcited by 835
  6. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 521
  7. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 183
  8. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 151

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026LambdaPO: A Lambda Style Policy Optimization for Reasoning Language Models
  2. 2026Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective
  3. 2025λ-GRPO: Unifying the GRPO Frameworks with Learnable Token Preferences
  4. 2025Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning
  5. 2025DGRO: Enhancing LLM Reasoning via Exploration-Exploitation Control and Reward Variance Management
  6. 2025COPO: Consistency-Aware Policy Optimization
  7. 2026P^2O: Joint Policy and Prompt Optimization
  8. 2025Unlocking Reasoning Capabilities in LLMs via Reinforcement Learning Exploration