# Marginally Correct Tool Caches Can Reverse Group-Normalized Policy Updates

arXiv: https://arxiv.org/abs/2609.26866 · HTML: https://mlsanity.com/paper/2609.26866/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 9 references that are in the map.
- First posted on arXiv 22 Sep 2026.

## Builds on (most cited first, 7 of 7)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradients](https://mlsanity.com/paper/2510.18924/) (2025)
- [TVCACHE: A Stateful Tool-Value Cache for Post-Training LLM Agents](https://mlsanity.com/paper/2602.10986/) (2026)
- [Abstention as an Action Can Kill Both the Reward Gradient and the KL Anchor: Collapse Law and Repair for Error-Penalized Reinforcement Learning](https://mlsanity.com/paper/2608.00301/) (2026)
- [CacheRL:Multi-Turn Tool-Calling Agents via Cached Rollouts and Hybrid Reward](https://mlsanity.com/paper/2606.14179/) (2026)
- [Are Verifier Errors Independent Within a GRPO Group? Evidence from Qwen2.5 Rollouts](https://mlsanity.com/paper/2609.06386/) (2026)

## Abstract (arXiv)

Tool-result caching reduces repeated execution in agent training, but also couples rollout randomness. We study a two-action model in which independent and shared execution preserve every rollout's conditional reward distribution. Despite this marginal agreement, sharing one stochastic result per group can reverse the expected group-normalized policy update. We derive an exact finite-group expression: against a constant alternative, the shared update follows the probability of winning minus the probability of losing, rather than the difference in expected reward. A Bernoulli specialization yields a wrong-direction region and a non-vanishing update-variance floor as group size grows. Centering without group standard-deviation scaling preserves the expected-return direction in this model, using an existing estimator control. Exhaustive finite sums verify 540 configurations and 3,240 estimator evaluations, with a separate ordered-sequence checker. An implementation audit reproduces the sharing path in a pinned, unmodified TVCache stack using 256 scripted rollouts. These results do not measure language-model training performance or refute TVCache's deterministic-output contract. They establish that marginal output validity alone cannot certify a stochastic cache as training-equivalent.