mlsanityData to 7 Oct 2026

DecisionBench: A Benchmark for Emergent Delegation in Long-Horizon Agentic Workflows

arXiv 2605.19099 ↗First posted 18 May 2026cs.AI cs.CL cs.MAcited by 2

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

We introduce DecisionBench, a benchmark substrate for emergent delegation in long-horizon agentic workflows. The substrate fixes a task suite (GAIA, tau-bench, BFCL multi-turn), a peer-model pool (11 models, 7 vendor families), a delegation interface (call_model plus an optional read_profile channel), a deterministic skill-annotation layer, and a multi-axis metric suite covering quality, cost, latency, delegation rate, routing fidelity-at-k, vendor self-preference, and a counterfactual-delegation ceiling. The substrate is agnostic to how peer information is generated or delivered, so learned routers, richer peer memories, adaptive profile construction, and multi-step delegation can all be evaluated against it. We characterize the substrate with a five-condition reference sweep on the full pool (n=23,375 task instances). Three benchmark-level findings emerge: (i) mean end-task quality is statistically indistinguishable across the four awareness conditions (|beta| <= 0.010, p >= 0.21), so quality-only evaluation would miss the orchestration signal; (ii) routing fidelity-at-1 ranges from 7.5% to 29.5% across conditions at near-equal mean quality, with delivery channel (on-demand tool vs. preloaded description) dominating description content; (iii) a counterfactual ceiling places perfect delegation 15-31 percentage points above measured performance on every suite, locating large unrealized headroom for future orchestration methods. We release the substrate, annotation layer, reference intervention suite, analysis pipeline, and 220 per-condition run archives.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 30 of its 45 references that are in the map. First posted on arXiv 18 May 2026.

Builds on

30 of 45 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains17 Jun
  2. 2024Mixture-of-Agents Enhances Large Language Model Capabilities7 Jun
  3. 2023The Rise and Potential of Large Language Model Based Agents: A Survey14 Sep

Other references · 27

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  6. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  7. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  8. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  9. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  10. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  11. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  12. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
15 more
  1. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  2. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  3. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  4. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  5. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  6. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  7. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  8. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
  9. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 550
  10. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  11. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  12. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 247
  13. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 222
  14. 2023Verbosity Bias in Preference Labeling by Large Language Modelscited by 89
  15. 2024MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Modelscited by 78

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026PerspectiveGap: A Benchmark for Multi-Agent Orchestration Promptingcited by 6
  2. 2026Position: Coding Benchmarks Are Misaligned with Agentic Software Engineeringcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2026Intelligent AI Delegation
  2. 2026Task-Aware Delegation Cues for LLM Agents
  3. 2026SearchSwarm: Towards Delegation Intelligence in Agentic LLMs for Long-Horizon Deep Research
  4. 2026MetaRoute-Bench: Evaluating Meta-Decision Policies for Agentic Workflow Routing
  5. 2026Safe Bilevel Delegation (SBD): A Formal Framework for Runtime Delegation Safety in Multi-Agent Systems
  6. 2026Learning Compositional Meta-Routing for Agentic Workflows: An Executable Benchmark
  7. 2026Uno-Orchestra: Parsimonious Agent Routing via Selective Delegation
  8. 2026AgentFugue: Agent Scaling for Long-Horizon Tasks through Collective Reasoning