mlsanityData to 7 Oct 2026

ChromaFlow: A Negative Ablation Study of Orchestration Overhead in Tool-Augmented Agent Evaluation

arXiv 2605.14102 ↗First posted 13 May 2026cs.AIcited by 0

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

Autonomous language-model agents increasingly combine planning, tool use, document processing, browsing, code execution, and verification loops. These capabilities make agent systems more useful, but they also introduce operational failure modes that are not visible from final accuracy alone. This report presents ChromaFlow, a tool-augmented autonomous reasoning framework built around planner-directed execution, specialized tool use, and telemetry-driven evaluation. We analyze ChromaFlow on GAIA 2023 Level-1 validation tasks under clean evaluation constraints. A frozen full Level-1 baseline achieved 29/53 correct answers, or 54.72%. A later recovery configuration with expanded orchestration achieved 27/53 correct answers, or 50.94%, while increasing tracebacks, timeout events, tool-failure mentions, token-log calls, and campaign-log cost estimates. Two randomized 20-task smoke evaluations produced 12/20 and 11/20 correct answers, showing that small diagnostic gains can be unstable across samples. The central result is therefore a negative ablation: more aggressive orchestration did not improve full-set performance and increased operational noise. A later strict-provider full-Level-1 diagnostic reached 30/53, or 56.60%, under explicit integrity controls, but at substantially higher token-log cost. The report argues that bounded planner escalation, deterministic extraction, evidence reconciliation, provider-health gates, and explicit run gates should be treated as first-order requirements for reliable autonomous agent evaluation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 9 of its 12 references that are in the map. First posted on arXiv 13 May 2026.

Builds on

9 of 12 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Toolformer: Language Models Can Teach Themselves to Use Tools9 Feb

Other references · 8

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  3. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  4. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  5. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  6. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  7. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 348
  8. 2026SoK: Agentic Skills -- Beyond Tool Use in LLM Agentscited by 120

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents
  2. 2026AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents
  3. 2026AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities
  4. 2026Designing for Doubt: The Case for Informed Abstention in Autonomous Agents
  5. 2026AgentEscapeBench: Evaluating Out-of-Domain Tool-Grounded Reasoning in LLM Agents
  6. 2026Jenius Agent: Towards Experience-Driven Accuracy Optimization in Real-World Scenarios
  7. 2026Test-time reasoning effort and unauthorized tool use in language-model agents: a prespecified equivalence study
  8. 2026Loud Failures, Quiet Failures: Fault Detection and Recovery in Tool-Using Language Model Agents