mlsanityData to 7 Oct 2026

SALT: When More Rollouts Don't Help in Group-Based Policy Optimization and How to Make Them Matter

arXiv 2606.05800 ↗First posted 4 Jun 2026cs.LGcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning with verifiable rewards (RLVR) often adopts GRPO-style group-relative updates, sampling multiple rollouts per prompt to construct normalized learning signals. However, merely increasing the number of rollouts does not reliably strengthen learning: under GRPO-style group normalization, per-rollout policy-gradient features can concentrate into a low-rank, signed geometry, causing substantial cancellation during aggregation and weakening the effective update. We address this failure mode with SALT, a Subspace-Adaptive geometry pLug-in componenT that uses sample-wise gradient geometry to reweight the coefficients of group-relative updates. SALT estimates a dominant shared subspace from the mini-batch Gram geometry, decomposes group-relative coefficients into shared and residual channels, and adaptively amplifies the residual channel when signed cancellation is severe. Across diverse reasoning-oriented RLVR benchmarks and model scales, SALT improves effective update geometry and performance without modifying the reward model or the rollout sampling procedure

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 32 of its 47 references that are in the map. First posted on arXiv 4 Jun 2026.

Builds on

32 of 47 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models11 Sep
  2. 2025EDGE-GRPO: Entropy-Driven GRPO with Guided Error Correction for Advantage Diversity29 Jul
  3. 2025Multi-Layer GRPO: Enhancing Reasoning and Self-Correction in Large Language Models5 Jun
  4. 2025Diversity-Aware Policy Optimization for Large Language Model Reasoning29 May
  5. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  6. 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning18 Apr
  7. 2023Let's Verify Step by Step31 May
  8. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 24

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2021Program Synthesis with Large Language Modelscited by 3,509
  7. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  8. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  9. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  10. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  11. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  12. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
12 more
  1. 2025Group Sequence Policy Optimizationcited by 788
  2. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  3. 2019Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Boundscited by 602
  4. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  5. 2023TRAK: Attributing Model Behavior at Scalecited by 273
  6. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  7. 2025Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variationcited by 44
  8. 2025On Entropy Control in LLM-RL Algorithmscited by 39
  9. 2024Does RLHF Scale? Exploring the Impacts From Data, Model, and Methodcited by 22
  10. 2025Fast Generalized Neural Tangent Kernel Statistics via Trace Estimationcited by 3
  11. 2021Zhestyatsky at SemEval-2021 Task 2: ReLU over Cosine Similarity for BERT Fine-tuningcited by 1
  12. 2025Group Policy Gradientcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Group Adaptive Clipping Policy Optimization
  2. 2026MC-GRPO: Median-Centered Group Relative Policy Optimization for Small-Rollout Reinforcement Learning
  3. 2026Train Less, Learn More: Adaptive Efficient Rollout Optimization for Group-Based Reinforcement Learning
  4. 2026Early Verdicts, Better Budgets: Sequential Adaptive Rollout Allocation for Compute-Efficient RLVR
  5. 2026Spend Your Rollouts Where It Counts: Rollout Allocation for Group-Based RL Post-Training
  6. 2026Where to Spend Rollouts: Hit-Utility Optimal Rollout Allocation for Group-Based RLVR
  7. 2025Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reuse
  8. 2026RRPO: Reference-Relative Policy Optimization with Stratified Conditional Rollouts