Beyond Leaderboards: Tokenomics of Agentic Small Language Model Ensembles
On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks
Abstract (arXiv)
As large language models (LLMs) move from standalone assistants into agentic workflows, evaluation must extend beyond scalar leaderboard accuracy to account for operational reliability, cost, latency, and token efficiency. We use an agentic ensemble of small language models (SLMs) with an SLM-judge-mediated feedback loop as a case study for such beyond-leaderboard evaluation. On the 541-prompt IFEval benchmark, the best ensemble achieves 97.34% strict prompt accuracy, exceeding the strongest standalone LLM baseline, gpt-5.4, by 5.81 percentage points while operating in a lower-cost regime. We then analyze the tokenomics and operational behavior behind this gain, including cost per sample, token composition, useful-output goodput, feedback-loop recovery, latency decomposition, and performance across instruction categories and constraint counts. Our results show that agentic SLM ensembles can trade additional test-time tokens and orchestration overhead for improved instruction-following fidelity, motivating multi-dimensional evaluation protocols for future agentic AI systems.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 5 of its 16 references that are in the map. First posted on arXiv 1 Oct 2026.
Builds on
5 of 16 references.
References · 5
- 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
- 2023GAIA: a benchmark for General AI Assistantscited by 1,128
- 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
- 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 137
- 2025Evaluating LLM Metrics Through Real-World Capabilitiescited by 18
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026UniACE: A Unified Framework for Evaluating LLM Agentic Capabilities
- 2025Small Language Models are the Future of Agentic AI
- 2025Small Language Models for Agentic Systems: A Survey of Architectures, Capabilities, and Deployment Trade offs
- 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents
- 2024Large Language Model Evaluation Via Multi AI Agents: Preliminary results
- 2026OneMillion-Bench: How Far are Language Agents from Human Experts?
- 2026When Agents Slow Down: Understanding LLM Agents' Test-Time Strategies via Elo-per-token Analysis
- 2026AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments