# When Is Smaller Greener? Right-Sizing Language Models for Agentic AI arXiv: https://arxiv.org/abs/2610.19345 · HTML: https://mlsanity.com/paper/2610.19345/ ## Key facts - No citing papers in the map yet (data through 14 Oct 2026). - Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. - Builds on 24 of its 58 references that are in the map. - First posted on arXiv 14 Oct 2026. ## Placed in - LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/) ## Builds on (most cited first, 24 of 24) - [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024) - [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019) - [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023) - [τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment](https://mlsanity.com/paper/2506.07982/) (2025) - [Small Language Models are the Future of Agentic AI](https://mlsanity.com/paper/2506.02153/) (2025) - [RouterBench: A Benchmark for Multi-LLM Routing System](https://mlsanity.com/paper/2403.12031/) (2024) - [AI Agents That Matter](https://mlsanity.com/paper/2407.01502/) (2024) - [Survey on Evaluation of LLM-based Agents](https://mlsanity.com/paper/2503.16416/) (2025) - [Power Hungry Processing: Watts Driving the Cost of AI Deployment?](https://mlsanity.com/paper/2311.16863/) (2023) - [Measuring AI Ability to Complete Long Software Tasks](https://mlsanity.com/paper/2503.14499/) (2025) - [AutoMix: Automatically Mixing Language Models](https://mlsanity.com/paper/2310.12963/) (2023) - [Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modeling](https://mlsanity.com/paper/2308.06077/) (2023) - [How Hungry is AI? Benchmarking Energy, Water, and Carbon Footprint of LLM Inference](https://mlsanity.com/paper/2505.09598/) (2025) - [The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs](https://mlsanity.com/paper/2509.09677/) (2025) - [The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization](https://mlsanity.com/paper/2505.06371/) (2025) - [Measuring the environmental impact of delivering AI at Google Scale](https://mlsanity.com/paper/2508.15734/) (2025) - [Energy Use of AI Inference, Efficiency Pathways, and Test-Time Scaling](https://mlsanity.com/paper/2509.20241/) (2025) - [ProgRouter: Online Progress-Guided Orchestration for Multi-Agent LLM Workflows under Quality-Cost Tradeoffs](https://mlsanity.com/paper/2608.25992/) (2026) - [Energy per Successful Goal: Goal-Level Energy Accounting for Agentic AI Systems](https://mlsanity.com/paper/2605.22883/) (2026) - [Cost-Utility Alignment in LLM Agent Trajectories:Profiling,Attribution,Diagnosis,Adaptation,and Evaluation](https://mlsanity.com/paper/2608.26195/) (2026) - [Toward Reliable Design of LLM-Enabled Agentic Workflows: Optimizing Latency-Reliability-Cost Tradeoffs](https://mlsanity.com/paper/2605.23929/) (2026) - [HugSelect: An Explainable Multi-Criteria Decision-Support Framework for foundation-model selection](https://mlsanity.com/paper/2608.08069/) (2026) - [Not All Tokens Are Equal: Inflation-Aware Routing for Agentic LLM Systems](https://mlsanity.com/paper/2608.13571/) (2026) - [How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making](https://mlsanity.com/paper/2609.01660/) (2026) ## Abstract (arXiv) AI agents use large language models (LLMs) to plan and act over multiple steps, yet models are usually compared per call rather than per completed task. This paper asks: which model should an agent use to complete tasks reliably while minimising energy, cost and time? We show that per-call comparisons can be misleading for agentic AI. Because a task succeeds only if all required steps succeed, small differences in per-step reliability can produce large differences in overall task success. Failed attempts also consume resources. We therefore evaluate models using energy, cost and time per successfully completed task, treating the completed task as the functional unit. On this basis, we develop a decision model for choosing LLMs in agentic workflows. The model applies to tasks with binary or graded outcomes and to settings with or without retries. We show that each resource has a crossover task length at which a smaller, less reliable model ceases to be the greener, cheaper or faster option. These crossovers divide tasks into three regimes: for short tasks smaller models are preferred, for long tasks larger models are preferred, and only between these extremes do preferences among objectives matter. We embed these results in a two-stage framework that separates efficiency from preference. A Pareto screen first removes models that are never competitive; a weighted decision is required only when a genuine trade-off remains. A pilot study of 15 open-weight models from five developers, ranging from 0.6B to 27B parameters, on two multi-step agent tasks with measured GPU energy supports the predicted structure. The model with the lowest energy per completed task changes with task length and task type, and observed crossover points generally match those predicted from per-step measurements. These findings provide a practical framework for right-sizing language models in agentic AI.