# MASPO: Unifying Gradient Utilization, Probability Mass, and Signal Reliability for Robust and Sample-Efficient LLM Reasoning

arXiv: https://arxiv.org/abs/2602.17550 · HTML: https://mlsanity.com/paper/2602.17550/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 23 of its 48 references that are in the map.
- First posted on arXiv 19 Feb 2026.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 23 of 23)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [Solving math word problems with process- and outcome-based feedback](https://mlsanity.com/paper/2211.14275/) (2022)
- [Reasoning with Exploration: An Entropy Perspective](https://mlsanity.com/paper/2506.14758/) (2025)
- [MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention](https://mlsanity.com/paper/2506.13585/) (2025)
- [Maximum Entropy RL (Provably) Solves Some Robust RL Problems](https://mlsanity.com/paper/2103.06257/) (2021)
- [Skywork Open Reasoner 1 Technical Report](https://mlsanity.com/paper/2505.22312/) (2025)
- [A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce](https://mlsanity.com/paper/2504.11343/) (2025)
- [Soft Adaptive Policy Optimization](https://mlsanity.com/paper/2511.20347/) (2025)
- [BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clipping](https://mlsanity.com/paper/2510.18927/) (2025)
- [Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs](https://mlsanity.com/paper/2505.12929/) (2025)
- [DCPO: Dynamic Clipping Policy Optimization](https://mlsanity.com/paper/2509.02333/) (2025)
- [Klear-Reasoner: Advancing Reasoning Capability via Gradient-Preserving Clipping Policy Optimization](https://mlsanity.com/paper/2508.07629/) (2025)
- [Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Reward](https://mlsanity.com/paper/2510.03222/) (2025)
- [Combining policy gradient and Q-learning](https://mlsanity.com/paper/1611.01626/) (2016)
- [CE-GPPO: Coordinating Entropy via Gradient-Preserving Clipping Policy Optimization in Reinforcement Learning](https://mlsanity.com/paper/2509.20712/) (2025)
- [When Importance Sampling Misallocates Credit: Asymmetric Ratios for Outcome-Supervised RL](https://mlsanity.com/paper/2510.06062/) (2025)
- [Who Stole Your Data? A Method for Detecting Unauthorized RAG Theft](https://mlsanity.com/paper/2510.07728/) (2025)
- [Maximizing Local Entropy Where It Matters: Prefix-Aware Localized LLM Unlearning](https://mlsanity.com/paper/2601.03190/) (2026)

## Built on by (most cited first, 11 of 11)

- [SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting](https://mlsanity.com/paper/2604.10688/) (2026)
- [Air-Know: Arbiter-Calibrated Knowledge-Internalizing Robust Network for Composed Image Retrieval](https://mlsanity.com/paper/2604.19386/) (2026)
- [ConeSep: Cone-based Robust Noise-Unlearning Compositional Network for Composed Image Retrieval](https://mlsanity.com/paper/2604.20358/) (2026)
- [TEMA: Anchor the Image, Follow the Text for Multi-Modification Composed Image Retrieval](https://mlsanity.com/paper/2604.21806/) (2026)
- [Group-Reflective Self-Distillation for Agentic Reinforcement Learning](https://mlsanity.com/paper/2607.28076/) (2026)
- [Learning from Contrasts: Synthesizing Reasoning Paths from Diverse Search Trajectories](https://mlsanity.com/paper/2604.11365/) (2026)
- [Beyond Semantic Relevance: Counterfactual Risk Minimization for Robust Retrieval-Augmented Generation](https://mlsanity.com/paper/2605.01302/) (2026)
- [IMAGINE: Adaptive Schema-Imagery Enhanced Composition for Composed Video Retrieval](https://mlsanity.com/paper/2606.08144/) (2026)
- [Gradients Know What Outcomes Don't: Unlocking Reinforcement Learning for LLM Reasoning with Gradient-Aligned Rewards](https://mlsanity.com/paper/2609.03342/) (2026)
- [Unlearnable, or Unmeasured? On the Reliability of Difficulty Labels in RLVR](https://mlsanity.com/paper/2609.40115/) (2026)
- [Placing Puzzle Pieces Where They Matter: A Question Augmentation Framework for Reinforcement Learning](https://mlsanity.com/paper/2604.15830/) (2026)

## Abstract (arXiv)

Existing Reinforcement Learning with Verifiable Rewards (RLVR) algorithms, such as GRPO, rely on rigid, uniform, and symmetric trust region mechanisms that are fundamentally misaligned with the complex optimization dynamics of Large Language Models (LLMs). In this paper, we identify three critical challenges in these methods: (1) inefficient gradient utilization caused by the binary cutoff of hard clipping, (2) insensitive probability mass arising from uniform ratio constraints that ignore the token distribution, and (3) asymmetric signal reliability stemming from the disparate credit assignment ambiguity between positive and negative samples. To bridge these gaps, we propose Mass-Adaptive Soft Policy Optimization (MASPO), a unified framework designed to harmonize these three dimensions. MASPO integrates a differentiable soft Gaussian gating to maximize gradient utility, a mass-adaptive limiter to balance exploration across the probability spectrum, and an asymmetric risk controller to align update magnitudes with signal confidence. Extensive evaluations demonstrate that MASPO serves as a robust, all-in-one RLVR solution, significantly outperforming baselines. Our code is available at: https://github.com/FlyTune/MASPO-RL.