mlsanityData to 14 Oct 2026

When Does Action Credit Need Updating?

arXiv 2609.29007 ↗First posted 24 Sep 2026cs.AI stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Tool-using agents are continually updated with new interaction data. After each policy update, however, previously estimated action credits may become stale. Recomputing them from scratch can require many additional tool calls and environment interactions, making repeated updates increasingly expensive. We ask a simple question: when does historical action credit actually need to be updated? Our key observation is that a change in action value does not necessarily imply a change in the decision. Historical credit can still be useful as long as policy-induced drift is too small to overturn the existing action ranking. Building on this idea, we introduce pairwise branch sensitivity to capture how strongly a policy update affects the downstream regions that distinguish two candidate actions. We then derive a first-order anchored credit-transport estimator that updates historical credit using old interventional trajectories, and propose a Decision-Sufficient Credit Gate (DSC-Gate) that chooses whether to reuse, transport, or resample credit. Experiments show that branch sensitivity explains credit drift substantially better than global policy distance. With sufficient historical data, credit transport reduces estimation error, while its benefit to decision making is concentrated on updates that affect action-distinguishing branches. On a fully independent test set, DSC-Gate changes mean regret by only +0.00004 relative to a gap-based gate while reducing mean new tool steps from 472 to 286, a 39.4% reduction. We observe the same pattern after a real tool-agent parameter update. Overall, our results show that agents do not need to recompute action credit after every policy update: much of the historical evidence can be reused or cheaply corrected, reducing the additional interaction required to keep action decisions up to date.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 33 of its 49 references that are in the map. First posted on arXiv 24 Sep 2026.

Builds on

33 of 49 references.

References · 33

  1. 2016Distribution-Free Predictive Inference For Regressioncited by 668
  2. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 591
  3. 2024ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilitiescited by 295
  4. 2015Doubly Robust Policy Evaluation and Optimizationcited by 158
  5. 2024StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Modelscited by 132
  6. 2023T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Stepcited by 93
  7. 2024ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stagescited by 82
  8. 2019Off-Policy Evaluation in Partially Observable Environmentscited by 66
  9. 2024Learning to Use Tools via Cooperative and Interactive Agentscited by 54
  10. 2025SMART: Self-Aware Agent for Tool Overuse Mitigationcited by 52
  11. 2024RoTBench: A Multi-Level Benchmark for Evaluating the Robustness of Large Language Models in Tool Learningcited by 47
  12. 2015Counterfactual Risk Minimization: Learning from Logged Bandit Feedbackcited by 33
21 more
  1. 2025CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agentscited by 33
  2. 2025Adaptive Tool Use in Large Language Models with Meta-Cognition Triggercited by 30
  3. 2025Procedural Environment Generation for Tool-Use Agentscited by 24
  4. 2024LLMs in the Imaginarium: Tool Learning through Simulated Trial and Errorcited by 19
  5. 2025Tool Preferences in Agentic LLMs are Unreliablecited by 18
  6. 2026The Confidence Dichotomy: Analyzing and Mitigating Miscalibration in Tool-Use Agentscited by 16
  7. 2023Safe Policy Improvement for POMDPs via Finite-State Controllerscited by 12
  8. 2025Rethinking Stateful Tool Use in Multi-Turn Dialogues: Benchmarks and Challengescited by 12
  9. 2021Variance Penalized On-Policy and Off-Policy Actor-Criticcited by 10
  10. 2021High-Confidence Off-Policy (or Counterfactual) Variance Estimationcited by 8
  11. 2023Human Choice Prediction in Language-based Persuasion Games: Simulation-based Off-Policy Evaluationcited by 8
  12. 2025Hephaestus: Improving Fundamental Agent Capabilities of Large Language Models through Continual Pre-Trainingcited by 6
  13. 2026Agent-Dice: Disentangling Knowledge Updates via Geometric Consensus for Agent Continual Learningcited by 6
  14. 2016Importance Sampling with Unequal Supportcited by 5
  15. 2025ToolRM: Towards Agentic Tool-Use Reward Modelingcited by 5
  16. 2026Rethinking the Role of Entropy in Optimizing Tool-Use Behaviors for Large Language Model Agentscited by 5
  17. 2023On the Sample Complexity of Vanilla Model-Based Offline Reinforcement Learning with Dependent Samplescited by 4
  18. 2022Counterfactual Learning with General Data-generating Policiescited by 3
  19. 2024Cross-Validated Off-Policy Evaluationcited by 3
  20. 2025Data-Efficient Safe Policy Improvement Using Parametric Structurecited by 2
  21. 2025Behaviour Policy Optimization: Provably Lower Variance Return Estimates for Off-Policy Reinforcement Learningcited by 2

Built on by

Not cited yet.