mlsanityData to 14 Oct 2026

The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspective

arXiv 2506.04301 ↗First posted 4 Jun 2025cs.LG cs.ARcited by 39

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

Large-language-model (LLM)-based AI agents have recently showcased impressive versatility by employing dynamic reasoning, an adaptive, multi-step process that coordinates with external tools. This shift from static, single-turn inference to agentic, multi-turn workflows broadens task generalization and behavioral flexibility, but it also introduces serious concerns about system-level cost, efficiency, and sustainability. This paper presents the first comprehensive system-level analysis of AI agents, quantifying their resource usage, latency behavior, energy consumption, and datacenter-wide power consumption demands across diverse agent designs and test-time scaling strategies. We further characterize how AI agent design choices, such as few-shot prompting, reflection depth, and parallel reasoning, impact accuracy-cost tradeoffs. Our findings reveal that while agents improve accuracy with increased compute, they suffer from rapidly diminishing returns, widening latency variance, and unsustainable infrastructure costs. Through detailed evaluation of representative agents, we highlight the profound computational demands introduced by AI agent workflows, uncovering a looming sustainability crisis. These results call for a paradigm shift in agent design toward compute-efficient reasoning, balancing performance with deployability under real-world constraints.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 54 of its 104 references that are in the map. First posted on arXiv 4 Jun 2025.

Builds on

54 of 104 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024AI Metropolis: Scaling Large Language Model-based Multi-Agent Simulation with Out-of-order Execution5 Nov
  2. 2023An LLM Compiler for Parallel Function Calling7 Dec
  3. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models6 Oct

Other references · 51

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  9. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  10. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  11. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
38 more
  1. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
  2. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  3. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  4. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  5. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  6. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  7. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  8. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  9. 2025s1: Simple test-time scalingcited by 1,341
  10. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  11. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  12. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  13. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  14. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  15. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 886
  16. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  17. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  18. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  19. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  20. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  21. 2025Small Language Models are the Future of Agentic AIcited by 270
  22. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 252
  23. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
  24. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 223
  25. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 208
  26. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 203
  27. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 163
  28. 2024Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Modelscited by 163
  29. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 148
  30. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 126
  31. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 125
  32. 2025Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuningcited by 123
  33. 2025Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoningcited by 119
  34. 2024LLM Pruning and Distillation in Practice: The Minitron Approachcited by 82
  35. 2025Autellix: An Efficient Serving Engine for LLM Agents as General Programscited by 58
  36. 2021Carbon-Aware Computing for Datacenterscited by 41
  37. 2024Teola: Towards End-to-End Optimization of LLM-based Applicationscited by 32
  38. 2023Clover: Toward Sustainable AI with Carbon-Aware Machine Learning Inference Servicecited by 17

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 28.

39 papers.

Sort:

By area:

  1. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 170
  2. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  3. 2025What Is Your AI Agent Buying? Evaluation, Biases, Model Dependence, & Emerging Implications for Agentic E-Commercecited by 22
  4. 2025Murakkab: Resource-Efficient Agentic Workflow Orchestration in Cloud Platformscited by 18
  5. 2026AgentArk: Distilling Multi-Agent Intelligence into a Single LLM Agentcited by 13
  6. 2026AgentCgroup: Understanding and Controlling OS Resources of AI Agentscited by 13
  7. 2026Inference-Time Budget Control for LLM Search Agentscited by 13
  8. 2026SWE-Replay: Efficient Test-Time Scaling for Software Engineering Agentscited by 12
  9. 2026Spend Less, Reason Better: Budget-Aware Value Tree Search for LLM Agentscited by 11
  10. 2025AutoTool: Efficient Tool Selection for Large Language Model Agentscited by 10
  11. 2025Routine: A Structural Planning Framework for LLM Agent System in Enterprisecited by 9
  12. 2026ICaRus: Identical Cache Reuse for Efficient Multi Model Inferencecited by 8
27 more
  1. 2025SLR: Automated Synthesis for Scalable Logical Reasoningcited by 6
  2. 2026AgentStop: Terminating Local AI Agents Early to Save Energy in Consumer Devicescited by 6
  3. 2026From LLM Inference to Agentic Workloads: Characterization and Implications for Serving Systemscited by 5
  4. 2025Towards Understanding, Analyzing, and Optimizing Agentic AI Execution: A CPU-Centric Perspectivecited by 4
  5. 2026EZBlender: Efficient 3D Editing with Plan-and-ReAct Agentcited by 4
  6. 2026PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Servingcited by 4
  7. 2026From Logic Monopoly to Social Contract: Separation of Power and the Institutional Foundations for Autonomous Agent Economiescited by 3
  8. 2026Characterization of Multi-Model Agentic AI Systems on General Tasks via Trace-Driven Simulationcited by 3
  9. 2026Specialize Roles, Mix Deployments: Pushing the Cost-Accuracy Frontier of LLM Agent Teamscited by 2
  10. 2026Architectural Implications of Agentic AI Workflowscited by 2
  11. 2026Cost-Utility Alignment in LLM Agent Trajectories:Profiling,Attribution,Diagnosis,Adaptation,and Evaluationcited by 2
  12. 2025ORION: Teaching Language Models to Reason Efficiently in the Language of Thoughtcited by 1
  13. 2026KAIROS: Stateful, Context-Aware Power-Efficient Agentic Inference Servingcited by 1
  14. 2026The Sim-to-Real Gap of Foundation Model Agents: A Unified MDP Perspectivecited by 1
  15. 2026WhiFlash: Accelerating Speculative Decoding with Token-Level Cross-Paradigm Routingcited by 1
  16. 2025DevNous: An LLM-Based Multi-Agent System for Grounding IT Project Management in Unstructured Conversationcited by 0
  17. 2025The Meta-Prompting Protocol: Orchestrating LLMs via Adversarial Feedback Loopscited by 0
  18. 2026Tessera: Unlocking Heterogeneous GPUs through Kernel-Granularity Disaggregationcited by 0
  19. 2026Same Voice, Different Lab: On the Homogenization of Frontier LLM Personalitiescited by 0
  20. 2026LLMSYS-HPOBench: Hyperparameter Optimization Benchmark Suite for Real-World LLM Systemscited by 0
  21. 2026Certificates without Electrons? Theory and Evidence on Impacts from AI-Driven Power Demandcited by 0
  22. 2026Coordinated Networking for On-Device Agent-Augmented Real-Time Communicationcited by 0
  23. 2026Resource Constraints and Performance in Agentic AI Systemscited by 0
  24. 2026Not All AI Agents Are Equal: Characterizing Resource and Performance Dynamicscited by 0
  25. 2026Hot-Cold Tiering of HBM and High Bandwidth Flash for Agentic LLM Servingcited by 0
  26. 2026Beyond Token Savings: A Systematic Study of Context Compression in LLM Agentscited by 0
  27. 2026LatentSift: Policy-State Filtering for Token-Efficient Verification of Software Engineering Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Multi-Agent Reasoning Improves Compute Efficiency: Pareto-Optimal Test-Time Scaling
  2. 2026Rethinking Inference-Time Scaling in Local Computer-Use Agents: Failure Modes and Compute Tradeoffs
  3. 2026AI Agent Systems: Architectures, Applications, and Evaluation
  4. 2026Agent Memory: Characterization and System Implications of Stateful Long-Horizon Workloads
  5. 2025Towards Efficient Agents: A Co-Design of Inference Architecture and System
  6. 2026AgentSLABench: Evaluating and Benchmarking Agentic Systems Under Resource Constraints
  7. 2026MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI
  8. 2025Learning When to Plan: Efficiently Allocating Test-Time Compute for LLM Agents