mlsanityData to 7 Oct 2026

The Replay Gap: Static Evaluation of Model Switching in LLM Agents Scores the Wrong World

arXiv 2608.08239 ↗First posted 8 Aug 2026cs.LG cs.CLcited by 7

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

LLM routers promise efficiency by matching each request to the cheapest adequate model, and are increasingly applied per step inside multi-step agents. Yet agentic routers are evaluated like single-turn routers: by replaying logged trajectories and substituting another model's recorded outputs, assuming the rest of the trajectory is unaffected. We test this assumption with branching rollouts: we fork live SWE-bench agent trajectories at controlled points, rebuild the environment, continue each fork with a different model, and compare against same-model control forks that isolate sampling and replay noise. Across six paired runs (~900 rollouts), swaps exceed their matched control floors by +0.25 to +0.66 normalized edit distance (multiplicity-corrected CIs exclude zero), rewriting 61-94% of post-fork actions; 74-77% of early swaps diverge at the first post-fork action, versus 6-35% of controls, leaving only 3% of replayed states valid. Divergence decreases with fork depth in both directions. All five outcome flips we observe occur in swap arms, upgrades rescuing unsolved instances and a downgrade losing the sole solve, and zero occur across 359 control forks. Scoring these same swaps with a log-stitching replay evaluator, replay mispredicts every success-relevant outcome call and predicts patches with 0.00-0.11 similarity to reality. Auditing the noise floor, temperature-0 "determinism" is configuration-dependent: FP8-served controls diverge on over 90% of forks while AWQ-served ones remain near-identical; and under tight budgets the stronger model more often exhausts its steps without submitting. Replay-based benchmarks score the wrong world for agentic routing; we release our harness and all trajectories.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 19 of its 22 references that are in the map. First posted on arXiv 8 Aug 2026.

Builds on

19 of 22 references.

References · 19

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  4. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  5. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  6. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 354
  7. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  8. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  9. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 222
  10. 2023AutoMix: Automatically Mixing Language Modelscited by 116
  11. 2025Universal Model Routing for Efficient LLM Inferencecited by 80
  12. 2025Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via Reinforcement Learningcited by 64
7 more
  1. 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 58
  2. 2025RouterArena: An Open Platform for Comprehensive Comparison of LLM Routerscited by 19
  3. 2026Step-level Optimization for Efficient Computer-use Agentscited by 7
  4. 2026ODAR: Principled Adaptive Routing for LLM Reasoning via Active Inferencecited by 4
  5. 2026TwinRouterBench: Fast Static and Live Dynamic Evaluation for Realistic Agentic LLM Routingcited by 4
  6. 2026Switchcraft: AI Model Router for Agentic Tool Callingcited by 3
  7. 2026RouteJudge: An Open Platform for Reproducible and Preference-Aware LLM Routingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026Engineering Reliable Commit Gates for Agentic AI: Cost-Aware Verification Portfolios under Common-Mode Data Failurescited by 1
  2. 2026Influence Is Not Authority: When Causal Guardrail Signals Make Legitimate Tool Use Look Like an Attack in Tool-Using LLM Agentscited by 0
  3. 2026Reach or Solve? Attributing Agentic RL Gains with Checkpoint Handoffscited by 0
  4. 2026What Does a Skill Actually Do? Estimands and Evaluation Validity for Tool and Skill Use in LLM Agents: A Critical Reviewcited by 0
  5. 2026Same Winners, Different Success Rates: Evaluating How LLM Agents Recover from Failurescited by 0
  6. 2026What Can Component-Replacement Evidence Establish? A Critical Scoping Review of Local Decisions in LLM Agentscited by 0
  7. 2026False Floors: LLM Safety Routing Evaluations Break Under Distribution Shiftcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Chronicle: Cut-Point Replay for Regression Testing of LLM Agents
  2. 2025SABER: Small Actions, Big Errors -- Safeguarding Mutating Steps in LLM Agents
  3. 2026Maintaining Benchmarks Against Increasingly Capable Agents: Detection and Remediation of Unearned Passes
  4. 2026The Handoff Tax: Continuing Non-Native Trajectories in LLM Agents
  5. 2026RouteGuard: Certifying Routing Gain in LLM Multi-Agent Systems When Complementarity Is Not Enough
  6. 2026Autonomous Topology Mutation: Safe Runtime Restructuring for Multi-Agent LLM Systems with Capability, State, and Shadow Invariants
  7. 2026BenchTrace: A Benchmark for Testing Reflection Ability and Controlled Evolution in LLM Agents
  8. 2026NetAgentBench: A State-Centric Benchmark for Evaluating Agentic Network Configuration