mlsanityData to 14 Oct 2026

Position: The Real Barrier to LLM Agent Usability is Agentic ROI

arXiv 2505.17767 ↗First posted 23 May 2025cs.CLcited by 12

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Large Language Model (LLM) agents represent a promising shift in human-AI interaction, moving beyond passive prompt-response systems to autonomous agents capable of reasoning, planning, and goal-directed action. While LLM agents are technically capable of performing a broad range of tasks, not all of these capabilities translate into meaningful usability. This position paper argues that the central question for LLM agent usability is no longer whether a task can be automated, but whether it delivers sufficient Agentic Return on Investment (Agentic ROI). Agentic ROI reframes evaluation from raw performance to a holistic, utility-driven perspective, guiding when, where, and for whom LLM agents should be deployed. Despite widespread application in high-ROI tasks like coding and scientific research, we identify a critical usability gap in mass-market, everyday applications. To address this, we propose a zigzag developmental trajectory: first scaling up to improve information gain and time savings, then scaling down to reduce cost. We present a strategic roadmap across these phases to make LLM agents truly usable, accessible, and scalable in real-world applications.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 103 of its 152 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

103 of 152 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025A Survey of AI Agent Protocols23 Apr
  2. 2025AutoPDL: Automatic Prompt Optimization for LLM Agents6 Apr
  3. 2025Agent models: Internalizing Chain-of-Action Generation into Reasoning models9 Mar
  4. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks18 Dec
  5. 2024Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistance16 Oct
  6. 2024FlowBench: Revisiting and Benchmarking Workflow-Guided Planning for LLM-based Agents21 Jun
  7. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Tasks4 Jun
  8. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challenges21 Jan
  9. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security10 Jan
  10. 2024Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospects7 Jan
  11. 2023A Zero-Shot Language Agent for Computer Control with Structured Reflection12 Oct
  12. 2023The Rise and Potential of Large Language Model Based Agents: A Survey14 Sep
4 more
  1. 2023A Survey on Large Language Model based Autonomous Agents22 Aug
  2. 2023ExpeL: LLM Agents Are Experiential Learners20 Aug
  3. 2023AgentBench: Evaluating LLMs as Agents7 Aug
  4. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework1 Aug

Other references · 87

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  7. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  8. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  9. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  10. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  11. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
  12. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
38 more
  1. 2016Concrete Problems in AI Safetycited by 1,642
  2. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  3. 2025s1: Simple test-time scalingcited by 1,341
  4. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  5. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  6. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  7. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  8. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  9. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  10. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  11. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  12. 2023ChatDev: Communicative Agents for Software Developmentcited by 919
  13. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  14. 2022Towards Reasoning in Large Language Models: A Surveycited by 596
  15. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
  16. 2018VirtualHome: Simulating Household Activities via Programscited by 415
  17. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 415
  18. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 401
  19. 2025Accelerating scientific discovery with Co-Scientistcited by 401
  20. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 386
  21. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 384
  22. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 364
  23. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 363
  24. 2024Alignment faking in large language modelscited by 360
  25. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 336
  26. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 327
  27. 2025DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environmentscited by 286
  28. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 284
  29. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  30. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 282
  31. 2025TTRL: Test-Time Reinforcement Learningcited by 275
  32. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 256
  33. 2023Auto-GPT for Online Decision Making: Benchmarks and Additional Opinionscited by 255
  34. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 239
  35. 2024ToolACE: Winning the Points of LLM Function Callingcited by 216
  36. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 180
  37. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 170
  38. 2025ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learningcited by 161

50 most cited of 87 All 87 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 5.

12 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2024Practical Considerations for Agentic LLM Systems
  2. 2025LLM-Based Human-Agent Collaboration and Interaction Systems: A Survey
  3. 2024Towards better Human-Agent Alignment: Assessing Task Utility in LLM-Powered Applications
  4. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Users
  5. 2025Agentic Large Language Models, a survey
  6. 2025Generalizability of Large Language Model-Based Agents: A Comprehensive Survey
  7. 2025Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agents
  8. 2026AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments