Luck Is Not Skill: When Do Paired Rollouts Help Group-Relative RL of LLM Agents?
On the map: LLM agents and tool use → Reinforcement learning for LLM agents
Abstract (arXiv)
Group-relative reinforcement learning compares rollouts of the same prompt, but independent environment noise can obscure these comparisons. We study paired rollouts, which share an event-keyed noise schedule within each group while preserving each rollout's marginal distribution. Pairing removes the between-schedule component of reward-contrast variance, but need not reduce gradient variance. For one-sided grader noise, we derive an exact condition for reduction and give a counterexample in which reward contrasts improve while gradient variance increases. A controlled study trains a 2B tool-use agent under tool faults and grader flips, with three seeds per design. The protocol was registered with a disclosed, previously completed pilot. Under tool faults, pairing improves final noisy-test success by +5.1 percentage points on average, with all three seed differences positive, but misses the registered learning-curve criterion. The criterion is also missed under grader flips: the validation-AUC difference is +0.003 (95% interval [-0.029, +0.033]). A gradient probe on eight distinct checkpoints from two fault-trained trajectories finds lower mean-centered covariance traces under both noise types: 21 to 30% for grader flips and 40 to 63% for tool faults. These finite-sample measurements support the variance mechanism without establishing a general learning-speed benefit. The results distinguish improving reward comparisons, reducing estimator variance, and improving learning.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 16 of its 27 references that are in the map. First posted on arXiv 21 Sep 2026.
Builds on
16 of 27 references.
Closest ancestors · 1
Same direction, newest first
Other references · 15
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
- 2015Trust Region Policy Optimizationcited by 2,580
- 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
- 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
- 2025Spurious Rewards: Rethinking Training Signals in RLVRcited by 209
- 2018Reinforcement Learning with Perturbed Rewardscited by 79
- 2026AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Conditioncited by 23
- 2013PEGASUS: A Policy Search Method for Large MDPs and POMDPscited by 22
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradients
- 2026MC-GRPO: Median-Centered Group Relative Policy Optimization for Small-Rollout Reinforcement Learning
- 2026PAIR: Pairwise-Aware Inclusion Reweighting for Adaptive Rollout Allocation in RLVR
- 2026Spend Your Rollouts Where It Counts: Rollout Allocation for Group-Based RL Post-Training
- 2026Leveraging Error Diversity in Group Rollouts for Reinforcement Learning
- 2026Beyond Policy Rollouts: Embedding-Space Directional Exploration with Anchored Noise for LLM RLVR
- 2026SALT: When More Rollouts Don't Help in Group-Based Policy Optimization and How to Make Them Matter
- 2026Double-Rollout Reinforcement Learning with Sampling-based Reward Estimation