mlsanityData to 14 Oct 2026

READY or Not: Reliable Enterprise Agent Deployment

arXiv 2609.02095 ↗First posted 2 Sep 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

An AI agent can perform well on benchmarks and still be unsuitable for deployment. Existing AI-agent benchmarks measure whether an agent can complete realistic professional work, whereas enterprise deployment asks a different question: whether an agent can meet a required reliability level, under acceptable human oversight, and at tolerable cost. We introduce Reliable Enterprise Agent Deployment (READY), a framework for qualifying AI agents for deployment on enterprise workflows. READY preserves each workflow's own definition of successful execution while applying a common qualification procedure. Given an agent, a workflow, and a class of candidate oversight policies, READY measures the reliability and operating cost of the human-AI system, selects the minimum-cost policy that satisfies a specified reliability target, and statistically qualifies it on held-out cases. The resulting deployment profile characterizes the supported operating point: reliability, human-oversight burden, and cost. READY is implemented as an open testbed that decouples workflow specification, execution, evaluation, and qualification, and runs on existing agent-evaluation infrastructure. In an end-to-end clinical-audit case study spanning 16 agent systems and 750 cases, READY reveals differences hidden by autonomous performance: two systems separated by only 0.3 points in autonomous accuracy (72.8% vs. 72.5%) require 39.2% versus 29.6% human review, respectively, to qualify at the same 76% reliability target under the evaluated oversight policy. READY thus shifts enterprise agent evaluation from how well can the agent perform the work? to under what conditions, and at what cost, can it be reliably deployed? By making those conditions explicit and statistically testable, READY provides a basis for comparing agent systems, setting oversight requirements, and making evidence-based deployment decisions.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 36 of its 133 references that are in the map. First posted on arXiv 2 Sep 2026.

Builds on

36 of 133 references.

References · 36

  1. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  2. 2016Concrete Problems in AI Safetycited by 1,642
  3. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  4. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  5. 2017Selective Classification for Deep Neural Networkscited by 852
  6. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 792
  7. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 703
  8. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 647
  9. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 627
  10. 2026Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfacescited by 585
  11. 2025Humanity's Last Examcited by 491
  12. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
24 more
  1. 2018AI safety via debatecited by 400
  2. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Taskscited by 326
  3. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agentscited by 306
  4. 2025DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agentscited by 248
  5. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 225
  6. 2024AI Agents That Mattercited by 219
  7. 2025GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Taskscited by 158
  8. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 157
  9. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 152
  10. 2018Supervising strong learners by amplifying weak expertscited by 145
  11. 2023Conformal Prediction with Large Language Models for Multi-Choice Question Answeringcited by 139
  12. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 121
  13. 2025xbench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World Evaluationscited by 120
  14. 2021Towards a Science of Human-AI Decision Making: A Survey of Empirical Studiescited by 86
  15. 2024WorkBench: a Benchmark Dataset for Agents in a Realistic Workplace Settingcited by 53
  16. 2026Agents' Last Examcited by 31
  17. 2025DRBench: A Realistic Benchmark for Enterprise Deep Researchcited by 22
  18. 2025Language Models Prefer What They Know: Relative Confidence Estimation via Confidence Preferencescited by 13
  19. 2024Understanding Likelihood Over-optimisation in Direct Alignment Algorithmscited by 7
  20. 2024Functional-level Uncertainty Quantification for Calibrated Fine-tuning on LLMscited by 6
  21. 2024Coverage-Constrained Human-AI Cooperation with Multiple Expertscited by 5
  22. 2026Designing Benchmarks for Knowledge Workcited by 5
  23. 2026AgentBeats: Agentifying Agent Assessment for Openness, Standardization, and Reproducibilitycited by 2
  24. 2026CliniCARE-Bench: Clinical Calibrated Audit of Medical Reasoning in EHRcited by 2

Built on by

Not cited yet.