mlsanityData to 14 Oct 2026

Measuring Agents in Production

arXiv 2512.04123 ↗First posted 2 Dec 2025cs.CY cs.AI cs.LGcited by 37

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

LLM-based agents already operate in production across many industries, yet we lack an understanding of what technical methods make deployments successful. We present the first systematic study of Measuring Agents in Production, MAP, using first-hand data from agent developers. We conducted 20 case studies via in-depth interviews and surveyed 86 deployed systems practitioners across 26 domains. We investigate why organizations build agents, how they build them, how they evaluate them, and their top development challenges. Our study finds that production agents are built using simple, controllable approaches: 68% execute at most 10 steps before human intervention, 70% rely on prompting off-the-shelf models instead of weight tuning, and 74% depend primarily on human evaluation. Reliability (consistent correct behavior over time) remains the top development challenge, which practitioners currently address through systems-level design. MAP documents the current state of production agents, providing the research community with visibility into deployment realities and underexplored research avenues.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 42 of its 119 references that are in the map. First posted on arXiv 2 Dec 2025.

Builds on

42 of 119 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025Evaluation and Benchmarking of LLM Agents: A Survey29 Jul
  2. 2025Agentic Large Language Models, a survey29 Mar
  3. 2025Survey on Evaluation of LLM-based Agents20 Mar
  4. 2025Multi-Agent Coordination across Diverse Applications: A Survey20 Feb
  5. 2025Agent-R: Training Language Model Agents to Reflect via Iterative Self-Training20 Jan
  6. 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMs10 Jan
  7. 2024AgentBank: Towards Generalized LLM Agents via Fine-Tuning on 50000+ Interaction Trajectories10 Oct
  8. 2024Minstrel: Structural Prompt Generation with Multi-Agents Coordination for Non-AI Experts20 Sep
  9. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improve25 Jul
  10. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains17 Jun
  11. 2024The Landscape of Emerging AI Agent Architectures for Reasoning, Planning, and Tool Calling: A Survey17 Apr
  12. 2024A Survey on Context-Aware Multi-Agent Systems: Techniques, Challenges and Future Directions3 Feb
3 more
  1. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challenges21 Jan
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Survey14 Sep
  3. 2023A Survey on Large Language Model based Autonomous Agents22 Aug

Other references · 27

  1. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  2. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  3. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  4. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,071
  5. 2025AlphaEvolve: A coding agent for scientific and algorithmic discoverycited by 867
  6. 2025Why Do Multi-Agent LLM Systems Fail?cited by 604
  7. 2025Agentic Context Engineering: Evolving Contexts for Self-Improving Language Modelscited by 398
  8. 2025An Illusion of Progress? Assessing the Current State of Web Agentscited by 174
  9. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 131
  10. 2025LlamaFirewall: An open source guardrail system for building secure AI agentscited by 127
  11. 2025MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learningcited by 86
  12. 2024A Complete Survey on LLM-based AI Chatbotscited by 71
15 more
  1. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 67
  2. 2024EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilitiescited by 52
  3. 2025ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Taskscited by 51
  4. 2025SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agentcited by 48
  5. 2024DrugAgent: Automating AI-aided Drug Discovery Programming through LLM Multi-Agent Collaborationcited by 44
  6. 2024Coevolving with the Other You: Fine-Tuning LLM with Sequential Cooperative Multi-Agent Reinforcement Learningcited by 33
  7. 2025Curie: Toward Rigorous and Automated Scientific Experimentation with AI Agentscited by 33
  8. 2024Ruffle&Riley: Insights from Designing and Evaluating a Large Language Model-Based Conversational Tutoring Systemcited by 26
  9. 2024From Lived Experience to Insight: Unpacking the Psychological Risks of Using AI Conversational Agentscited by 25
  10. 2025Code Researcher: Deep Research Agent for Large Systems Code and Commit Historycited by 20
  11. 2025PlanGEN: A Multi-Agent Framework for Generating Planning and Reasoning Trajectories for Complex Problem Solvingcited by 15
  12. 2025Toward Agentic AI: Generative Information Retrieval Inspired Intelligent Communications and Networkingcited by 9
  13. 2025Why Johnny Can't Use Agents: Industry Aspirations vs. User Realities with AI Agentscited by 5
  14. 2024Trusting Your AI Agent Emotionally and Cognitively: Development and Validation of a Semantic Differential Scale for AI Trustcited by 2
  15. 2025Ask Me Anything: Exploring children's attitudes toward an age-tailored AI-powered chatbotcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 36.

37 papers.

Sort:

By area:

  1. 2026AgencyBench: Benchmarking the Frontiers of Autonomous Agents in 1M-Token Real-World Contextscited by 28
  2. 2026How Well Does Agent Development Reflect Real-World Work?cited by 10
  3. 2025An Information Theoretic Perspective on Agentic System Designcited by 8
  4. 2026EnterpriseBench Corecraft: Training Generalizable Agents on High-Fidelity RL Environmentscited by 8
  5. 2026A Trace-Based Assurance Framework for Agentic AI Orchestration: Contracts, Testing, and Governancecited by 8
  6. 2026Understanding the Challenges in Iterative Generative Optimization with LLMscited by 8
  7. 2026Riemann-Bench: A Benchmark for Moonshot Mathematicscited by 8
  8. 2026MAESTRO: Multi-Agent Evaluation Suite for Testing, Reliability, and Observabilitycited by 7
  9. 2026Training Agents to Self-Report Misbehaviorcited by 6
  10. 2026The Hierarchy of Agentic Capabilities: Evaluating Frontier Models on Realistic RL Environmentscited by 5
  11. 2026Fantastic Adaptive Taxonomies and How to Use Themcited by 4
  12. 2026Evidence-Decision-Feedback: Theory-Driven Adaptive Scaffolding for LLM Agentscited by 3
25 more
  1. 2026ComplexConstraints and Beyond: Expert Rubrics for RLVRcited by 3
  2. 2026Agentics 2.0: Logical Transduction Algebra for Agentic Data Workflowscited by 2
  3. 2026Signals: Trajectory Sampling and Triage for Agentic Interactionscited by 2
  4. 2026SynAE: A Framework for Measuring the Quality of Synthetic Data for Tool-Calling Agent Evaluationscited by 2
  5. 2026The "I Don't Know" Filter: Enhancing Agentic Reliability in Function Callingcited by 2
  6. 2026How Do Practitioners Build SE Agents? Insights from a Mixed-Methods Studycited by 2
  7. 2026Agent Hacks Agent: Autoresearch for Production-Agent Red-Teamingcited by 2
  8. 2026PieArena: Ranking and Profiling Language Agents in Realistic Negotiation Scenarioscited by 1
  9. 2026Generating Expressive and Customizable Evals for Timeseries Data Analysis Agents with AgentFuelcited by 1
  10. 2026I must delete the evidence: AI Agents Explicitly Cover up Fraud and Violent Crimecited by 1
  11. 2026Effective and Efficient Context Retrieval via Partial Dependency Graph for Repository-Level Code Generationcited by 1
  12. 2026Exploring Robust Multi-Agent Workflows for Environmental Data Managementcited by 0
  13. 2026Low-Data Supervised Adaptation Outperforms Prompting for Cloud Segmentation Under Domain Shiftcited by 0
  14. 2026Do LLMs Need to See Everything? A Benchmark and Study of Failures in LLM-driven Smartphone Automation using Screentext vs. Screenshotscited by 0
  15. 2026Echo: Learning from Experience Data via User-Driven Refinementcited by 0
  16. 2026A Theory-Guided LLM Pedagogical Agent for STEM+C Scaffolding Without Over-Reliancecited by 0
  17. 2026Hidden in Plain Sight: Benchmarking Agent Safety Against Decomposition Attacks with DECOMPBENCHcited by 0
  18. 2026Toward Trustworthy Autonomous Science: A Two-Year Community Roadmapcited by 0
  19. 2026Relay-Bench: Evaluating LLMs on Multi-Domain Reasoning Chainscited by 0
  20. 2026OpScale: Operator-level Provisioning and Autoscaling for LLM Servingcited by 0
  21. 2026Using profiles of cognitive capability to assess AI suitability for workplace taskscited by 0
  22. 2026EnterpriseVal: Quantifying the Efficacy, Reliability and Value of Generative AI in the Enterprisecited by 0
  23. 2026From Alignment to Access Control: A Framework for GenAI Policy Enforcementcited by 0
  24. 2026The Silent Failures in Agentic Data Sciencecited by 0
  25. 2026Beyond Corrected Actions: Training LLM Agents on Textual Critiquescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Static Leaderboards: Predictive Validity for the Evaluation of LLM Agents
  2. 2026AlphaEval: Evaluating Agents in Production
  3. 2026Efficient Benchmarking in Production: A Study of an Evolving LLM Agent
  4. 2026How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making
  5. 2026Benchmarks are Not Enough: RAMP for Runtime Assessing of Agentic Models in Production Systems
  6. 2026Specialize Roles, Mix Deployments: Pushing the Cost-Accuracy Frontier of LLM Agent Teams
  7. 2024Evaluation-Driven Development and Operations of LLM Agents: A Process Model and Reference Architecture
  8. 2026AgentSWE: Can Coding Agents Build the Agent You Actually Want?