Position: The Real Barrier to LLM Agent Usability is Agentic ROI
On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks
Abstract (arXiv)
Large Language Model (LLM) agents represent a promising shift in human-AI interaction, moving beyond passive prompt-response systems to autonomous agents capable of reasoning, planning, and goal-directed action. While LLM agents are technically capable of performing a broad range of tasks, not all of these capabilities translate into meaningful usability. This position paper argues that the central question for LLM agent usability is no longer whether a task can be automated, but whether it delivers sufficient Agentic Return on Investment (Agentic ROI). Agentic ROI reframes evaluation from raw performance to a holistic, utility-driven perspective, guiding when, where, and for whom LLM agents should be deployed. Despite widespread application in high-ROI tasks like coding and scientific research, we identify a critical usability gap in mass-market, everyday applications. To address this, we propose a zigzag developmental trajectory: first scaling up to improve information gain and time savings, then scaling down to reduce cost. We present a strategic roadmap across these phases to make LLM agents truly usable, accessible, and scalable in real-world applications.
Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 103 of its 152 references that are in the map. First posted on arXiv 23 May 2025.
Builds on
103 of 152 references.
Closest ancestors · 16
Same direction, newest first
- 2025A Survey of AI Agent Protocols23 Apr
- 2025AutoPDL: Automatic Prompt Optimization for LLM Agents6 Apr
- 2025Agent models: Internalizing Chain-of-Action Generation into Reasoning models9 Mar
- 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks18 Dec
- 2024Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistance16 Oct
- 2024FlowBench: Revisiting and Benchmarking Workflow-Guided Planning for LLM-based Agents21 Jun
- 2024Chain of Agents: Large Language Models Collaborating on Long-Context Tasks4 Jun
- 2024Large Language Model based Multi-Agents: A Survey of Progress and Challenges21 Jan
- 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security10 Jan
- 2024Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospects7 Jan
- 2023A Zero-Shot Language Agent for Computer Control with Structured Reflection12 Oct
- 2023The Rise and Potential of Large Language Model Based Agents: A Survey14 Sep
4 more
Other references · 87
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
- 2025Qwen3 Technical Reportcited by 8,820
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
- 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
- 2022Training Compute-Optimal Large Language Modelscited by 3,019
- 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
- 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
- 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
38 more
- 2016Concrete Problems in AI Safetycited by 1,642
- 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
- 2025s1: Simple test-time scalingcited by 1,341
- 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
- 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
- 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
- 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
- 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
- 2023GAIA: a benchmark for General AI Assistantscited by 1,128
- 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
- 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
- 2023ChatDev: Communicative Agents for Software Developmentcited by 919
- 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
- 2022Towards Reasoning in Large Language Models: A Surveycited by 596
- 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
- 2018VirtualHome: Simulating Household Activities via Programscited by 415
- 2023AppAgent: Multimodal Agents as Smartphone Userscited by 415
- 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 401
- 2025Accelerating scientific discovery with Co-Scientistcited by 401
- 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 386
- 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 384
- 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 364
- 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 363
- 2024Alignment faking in large language modelscited by 360
- 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 336
- 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 327
- 2025DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environmentscited by 286
- 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 284
- 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
- 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 282
- 2025TTRL: Test-Time Reinforcement Learningcited by 275
- 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 256
- 2023Auto-GPT for Online Decision Making: Benchmarks and Additional Opinionscited by 255
- 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 239
- 2024ToolACE: Winning the Points of LLM Function Callingcited by 216
- 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 180
- 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 170
- 2025ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learningcited by 161
50 most cited of 87 All 87 (JSON)
Built on by
Citing papers per year: 2025: 7; 2026: 5.
12 papers.
- 2025A Survey of Context Engineering for Large Language Modelscited by 126
- 2026Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineeringcited by 74
- 2025TinyScientist: An Interactive, Extensible, and Controllable Framework for Building Research Agentscited by 8
- 2025AgentCompass: Towards Reliable Evaluation of Agentic Workflows in Productioncited by 6
- 2025PARSE: LLM Driven Schema Optimization for Reliable Entity Extractioncited by 6
- 2026SMMBench: A Benchmark for Source-Distributed Multimodal Agent Memorycited by 6
- 2026SkillMAS: Skill Co-Evolution with LLM-based Multi-Agent Systemcited by 5
- 2025AURA: An Agent Autonomy Risk Assessment Frameworkcited by 4
- 2025ParaCook: On Time-Efficient Planning for Multi-Agent Systemscited by 3
- 2026What Do LLM Agents Know About Their World? Task2Quiz: A Paradigm for Studying Environment Understandingcited by 3
- 2025Fast, Slow, and Tool-augmented Thinking for LLMs: A Reviewcited by 2
- 2026OR-Space: A Full-Lifecycle Workspace Benchmark for Industrial Optimization Agentscited by 2
- 2026OR-Space: A Full-Lifecycle Workspace Benchmark for Industrial Optimization Agents27 May 2026
- 2026SMMBench: A Benchmark for Source-Distributed Multimodal Agent Memory15 May 2026
- 2026SkillMAS: Skill Co-Evolution with LLM-based Multi-Agent System10 May 2026
- 2026Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering9 Apr 2026
- 2026What Do LLM Agents Know About Their World? Task2Quiz: A Paradigm for Studying Environment Understanding14 Jan 2026
- 2025AURA: An Agent Autonomy Risk Assessment Framework17 Oct 2025
- 2025ParaCook: On Time-Efficient Planning for Multi-Agent Systems13 Oct 2025
- 2025PARSE: LLM Driven Schema Optimization for Reliable Entity Extraction8 Oct 2025
- 2025TinyScientist: An Interactive, Extensible, and Controllable Framework for Building Research Agents8 Oct 2025
- 2025AgentCompass: Towards Reliable Evaluation of Agentic Workflows in Production18 Sep 2025
- 2025Fast, Slow, and Tool-augmented Thinking for LLMs: A Review17 Aug 2025
- 2025A Survey of Context Engineering for Large Language Models17 Jul 2025
Similar, not cited
Close in wording, no citation either way
- 2024Practical Considerations for Agentic LLM Systems
- 2025LLM-Based Human-Agent Collaboration and Interaction Systems: A Survey
- 2024Towards better Human-Agent Alignment: Assessing Task Utility in LLM-Powered Applications
- 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Users
- 2025Agentic Large Language Models, a survey
- 2025Generalizability of Large Language Model-Based Agents: A Comprehensive Survey
- 2025Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agents
- 2026AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments