# Process Advantage Signal Shaping: A Paradigm-Agnostic Middleware for Process-Supervised RL in LLM Reasoners

arXiv: https://arxiv.org/abs/2606.29296 · HTML: https://mlsanity.com/paper/2606.29296/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 19 of its 26 references that are in the map.
- First posted on arXiv 28 Jun 2026.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)
- LLM alignment / RLHF / preference optimisation / [Process reward models for reasoning](https://mlsanity.com/alignment/process-reward-models-for-reasoning/)
- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 19 of 19)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://mlsanity.com/paper/2409.12122/) (2024)
- [The Lessons of Developing Process Reward Models in Mathematical Reasoning](https://mlsanity.com/paper/2501.07301/) (2025)
- [Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe](https://mlsanity.com/paper/2604.13016/) (2026)
- [A Long Way to Go: Investigating Length Correlations in RLHF](https://mlsanity.com/paper/2310.03716/) (2023)
- [GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization](https://mlsanity.com/paper/2601.05242/) (2026)
- [Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation](https://mlsanity.com/paper/2602.12125/) (2026)
- [What Makes a Reward Model a Good Teacher? An Optimization Perspective](https://mlsanity.com/paper/2503.15477/) (2025)
- [Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Training](https://mlsanity.com/paper/2509.03403/) (2025)
- [FIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimization](https://mlsanity.com/paper/2603.19835/) (2026)
- [From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Models](https://mlsanity.com/paper/2604.09459/) (2026)
- [DRPO: Efficient Reasoning via Decoupled Reward Policy Optimization](https://mlsanity.com/paper/2510.04474/) (2025)
- [GRPO is Secretly a Process Reward Model](https://mlsanity.com/paper/2509.21154/) (2025)
- [Enhancing Large Language Model Reasoning with Reward Models: An Analytical Survey](https://mlsanity.com/paper/2510.01925/) (2025)
- [Beyond Uniform Credit: Causal Credit Assignment for Policy Optimization](https://mlsanity.com/paper/2602.09331/) (2026)
- [Tackling Length Inflation Without Trade-offs: Group Relative Reward Rescaling for Reinforcement Learning](https://mlsanity.com/paper/2603.10535/) (2026)
- [Pinpointing crucial steps: Attribution-based Credit Assignment for Verifiable Reinforcement Learning](https://mlsanity.com/paper/2510.08899/) (2025)

## Built on by (most cited first, 2 of 2)

- [A Continuous-Time Reinforcement Learning Framework for Fine-Tuning Discrete Diffusion Models](https://mlsanity.com/paper/2607.14522/) (2026)
- [SWE-MILE: Asynchronous Potential-Induced Milestone Credit Assignment for Long-Horizon Software Engineering Agents](https://mlsanity.com/paper/2609.32631/) (2026)

## Abstract (arXiv)

Dense process supervision supplies local feedback but leaves open how that feedback should become a policy advantage. We introduce PASS (Process Advantage Signal Shaping), an advantage-construction pipeline for group-relative policy optimization (GRPO). PASS combines separate reward-channel normalization and fusion, signal-defined chunks for credit accumulation, and normalization by the remaining chunk count. This design connects the scale of local feedback to the resolution and horizon over which it contributes to a policy update. We evaluate PASS using learned process rewards for mathematical reasoning and teacher-derived token signals for multi-hop question answering. On seven mathematics benchmarks, PASS improves average pass@1 by 4.62 points over outcome-only GRPO while reducing average response length by approximately 2%. On multi-hop QA with mean-centered normalization, PASS exceeds outcome-only training by 0.87 and 0.63 points for two distillation signals, with longer responses. Across both normalization schemes, PASS achieves the highest mean accuracy among the tested process-signal configurations and remains closest to outcome-only under absolute-maximum normalization. These findings show that channel handling, credit resolution, and accumulation should be considered jointly when incorporating dense supervision into policy optimization.