mlsanityData to 7 Oct 2026

Re-feeding Is Not Replaying: Measuring Replay Noise in Counterfactual Token-Credit Estimation

arXiv 2606.15621 ↗First posted 14 Jun 2026cs.LG cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Per-token counterfactual credit estimation asks which token in a language-model rollout caused the final answer to be right or wrong: cut the transcript at a pivot, substitute an alternative token, replay continuations, and compare outcomes. Published methods re-feed the transcript prefix as a fresh prompt, assuming this reproduces the state the model passed through during generation. We measure what that assumption costs on a stock inference engine, with a three-pass design: continuations resumed from the verified decode-time KV state, an identical second exact pass (a replica noise floor), and a re-feed pass. Across six configurations and three models (including a GRPO-trained checkpoint), at low-margin decision tokens, re-feeding changes the credit estimate at rates 14-28 percentage points above the replica floor (7-21pp under a treatment-independent conditioning; problem-clustered t = 2.9-6.4). Most changes are zero-boundary crossings of the quantized estimator rather than polarity reversals, and the perturbation is consistent with mean-zero, so averaged quantities are largely safe; but selection is not: a critical-token set chosen by thresholding |hatAt| under re-feed overlaps the exact-resume selection at Jaccard 0.34-0.90, versus a 0.63-0.96 replica ceiling. A causal confirmation closes the loop: under vLLM's batch-invariant kernels all three passes are identical on every measured channel, with both disagreement rates exactly zero. Replica passes themselves disagree on 9-23% of eligible estimates: single-sample credit measurements at decision tokens are unreliable under any replay. Settings were fixed in advance; exact-pass cache hits in the second campaign are instrumented (100% hit rate, 3,434 pivots); total compute was under 10 USD. We recommend that counterfactual credit studies resume decoder state or use batch-invariant kernels, and report a replica floor.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 18 of its 24 references that are in the map. First posted on arXiv 14 Jun 2026.

Builds on

18 of 24 references.

References · 18

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2024Qwen2.5 Technical Reportcited by 4,713
  5. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 505
  6. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
  7. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 168
  8. 2024The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinismcited by 113
  9. 2024Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capabilitycited by 80
  10. 2025Defeating the Training-Inference Mismatch via FP16cited by 62
  11. 2025TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modelingcited by 60
  12. 2025Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inferencecited by 47
6 more
  1. 2025Enhancing Large Language Model Reasoning via Selective Critical Token Fine-Tuningcited by 20
  2. 2026From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Modelscited by 19
  3. 2024Counterfactual Token Generation in Large Language Modelscited by 14
  4. 2026Exact Is Easier: Credit Assignment for Cooperative LLM Agentscited by 9
  5. 2026Beyond Uniform Credit: Causal Credit Assignment for Policy Optimizationcited by 7
  6. 2026Diagnosing Training Inference Mismatch in LLM Reinforcement Learning via a Zero-Mismatch Referencecited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Stage-Replay Divergence Follows the KV Cache: Fixed-Prefix Precision Controls and Bidirectional Cache Transplantationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Not All Tokens Deserve Equal Credit: Counterfactual Sensitivity Credit Reallocation for Long-CoT Reasoning
  2. 2026Feedback That Backfires: Why Small Language Model Agents Repeat the Call They Just Watched Fail
  3. 2026ARCA: Adapter-Residual Credit Assignment When Token Signals Degenerate
  4. 2026CREDO: Variance-Guided Rubric Evolution for Replay-Corrected Credit Assignment
  5. 2026Credit Without Ground Truth: Auditing Step-Level Credit Assignment in LLM Agents Against Executed Replay
  6. 2026Locally Confident, Globally Stuck: The Quality-Exploration Dilemma in Diffusion Language Models
  7. 2026PACT: From Credit Assignment to Critic Alignment
  8. 2026Phantom Gains: Auditing Self-Improvement Against a Measured Null