Agentic AI Systems Should Be Designed as Marginal Token Allocators
On the map: LLM agents and tool use → Agent memory systems
Abstract (arXiv)
This position paper argues that agentic AI systems should be designed and evaluated as marginal token allocation economies rather than as text generators priced by the unit. We follow a single request -- a developer asking a coding agent to fix a failing test -- through four economic layers that today are designed in isolation: a router that decides which model answers, an agent that decides whether to plan, act, verify, or defer, a serving stack that decides how to produce each token, and a training pipeline that decides whether the trace is worth learning from. We show that all four layers are solving the same first-order condition -- marginal benefit equals marginal cost plus latency cost plus risk cost -- with different index sets and different prices. The framing is deliberately minimal: we do not propose a complete theory of AI economics. But adopting marginal token allocation as the shared accounting object explains why systems that locally minimize tokens globally misallocate them, predicts a small set of recurring failure modes (over-routing, over-delegation, under-verification, serving congestion, stale rollouts, cache misuse), and points to a concrete research agenda in token-aware evaluation, autonomy pricing, congestion-priced serving, and risk-adjusted RL budgeting.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Agent memory systems, one of 16 directions in LLM agents and tool use. Builds on 28 of its 47 references that are in the map. First posted on arXiv 2 May 2026.
Builds on
28 of 47 references.
References · 28
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
- 2023Let's Verify Step by Stepcited by 3,789
16 more
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
- 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
- 2022Training Compute-Optimal Large Language Modelscited by 3,019
- 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
- 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
- 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
- 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
- 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
- 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
- 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
- 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 238
- 2024Efficient LLM Scheduling by Learning to Rankcited by 85
- 2024Efficiently Scaling LLM Reasoning with Certaindexcited by 58
- 2026SRT: Accelerating Reinforcement Learning via Speculative Rollout with Tree-Structured Cachecited by 5
- 2026OpenTinker: Separating Concerns in Agentic Reinforcement Learningcited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Token Economics for LLM Agents: A Dual-View Study from Computing and Economics
- 2026The Harness Effect: How Orchestration Design Sets the Token Economics of Enterprise Agentic AI
- 2026How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks
- 2026Can your AI agent be cheaper? Investigating the effects of task specifications on token spend in agentic coding tasks
- 2026Computing with Agentic Oracles
- 2026Beyond Token Savings: A Systematic Study of Context Compression in LLM Agents
- 2026Inference Economics of Enterprise Coding Agents: A Case Study of Cloud vs. On-Premise LLMs
- 2026pAI-Econ-claude: A Gated Human-in-the-Loop Multi-Agent Architecture for AI-Assisted Economic Theory Development