mlsanityData to 7 Oct 2026

AllocBench: Measuring Online Tool Allocation Capability in LLM Agents

arXiv 2607.23332 ↗First posted 25 Jul 2026cs.LGcited by 2

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

Creating a reusable tool is an investment: an agent pays a fixed cost now in exchange for the potential of future reuse. Therefore, a user should prefer an agent that creates a small number of highly reusable tools, rather than many one-offs. We introduce a paired benchmark that tests whether LLM agents exhibit conscious allocation behavior under a fixed budget in two contexts: an abstract text-based formulation and a code-construction task. We find that every frontier model we test---Claude Haiku, Claude Opus, GPT-5.4-mini, and GPT-5.6 Sol---acts near-optimally in the abstract framing but fails to transfer this ability to script-writing. Through further experiments, we identify the particular failure modes for each model. Notably, the first three models fail even when the scripts are not evaluated, while GPT-5.6 Sol stays selective under that weaker manipulation and collapses only at full construction. Furthermore, an open-source Qwen model policy-trained for abstract allocation generalizes this ability across held-out lexical variations, but sees no improvement at script allocation. Together, these results establish online tool allocation as a significant capability boundary, even for modern frontier models.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 26 of its 33 references that are in the map. First posted on arXiv 25 Jul 2026.

Builds on

26 of 33 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026Budget-Constrained Agentic Large Language Models: Intention-Based Planning for Costly Tool Use12 Feb
  2. 2024Tool Learning with Large Language Models: A Survey28 May
  3. 2023Large Language Models as Tool Makers26 May

Other references · 23

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  3. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  4. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolutioncited by 246
  5. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 243
  6. 2017Asymmetric Actor Critic for Image-Based Robot Learningcited by 202
  7. 2025Competitive Programming with Large Reasoning Modelscited by 115
  8. 2023CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Modelscited by 90
  9. 2024Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Real-World Complex Scenarioscited by 85
  10. 2024TroVE: Inducing Verifiable and Efficient Toolboxes for Solving Programmatic Taskscited by 70
  11. 2023LILO: Learning Interpretable Libraries by Compressing and Documenting Codecited by 46
  12. 2025Budget-Aware Tool Use Enables Effective Agent Scalingcited by 42
11 more
  1. 2024Do LLM Agents Have Regret? A Case Study in Online Learning and Gamescited by 33
  2. 2024ReGAL: Refactoring Programs to Discover Generalizable Abstractionscited by 32
  3. 2026Calibrate-Then-Act: Cost-Aware Exploration in LLM Agentscited by 16
  4. 2026Benchmarking LLM Tool-Use in the Wildcited by 15
  5. 2026BAGEN: Are LLM Agents Budget-Aware?cited by 12
  6. 2026Spend Less, Reason Better: Budget-Aware Value Tree Search for LLM Agentscited by 10
  7. 2026When Agents Commit Too Soon: Diagnosing Premature Commitment in LLM Agentscited by 9
  8. 2026Utility-Guided Agent Orchestration for Efficient LLM Tool Usecited by 4
  9. 2024Library Learning Doesn't: The Curious Case of the Single-Use "Library"cited by 3
  10. 2025When Greedy Wins: Emergent Exploitation Bias in Meta-Bandit LLM Trainingcited by 3
  11. 2026On Time, Within Budget: Constraint-Driven Online Resource Allocation for Agentic Workflowscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026When Stale Constraints Go Unchecked: Budgeted Verification Failures in Inherited Agent Memorycited by 3
  2. 2026Towards Reliable AI Data Scientists: Data Agents with Workflow Harnessescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Token Savings: A Systematic Study of Context Compression in LLM Agents
  2. 2026SkillCraft: Can LLM Agents Learn to Use Tools Skillfully?
  3. 2026How Consistent Are LLM Agents? Measuring Behavioral Reproducibility in Multi-Step Tool-Calling Pipelines
  4. 2026DOCSCHISEL: Adaptive Tool Documentation Optimization Framework for LLM Agents
  5. 2026Compression, structure, and executor capability: a controlled real-cost decomposition of language-model agent skill optimisation
  6. 2026AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling
  7. 2026ToolMenuBench: Benchmarking Tool-Menu Filtering Strategies for Reliable and Efficient LLM Agents
  8. 2026The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestration