EnterpriseVal: Quantifying the Efficacy, Reliability and Value of Generative AI in the Enterprise
On the map: LLM agents and tool use → Software engineering coding agents also fits: General LLM agent frameworks and benchmarks
Abstract (arXiv)
Frontier language models now produce professional deliverables that expert graders judge to match human work on a substantial share of economically valuable tasks, yet most enterprise GenAI initiatives fail to show a measurable business effect and a large fraction of agentic projects are expected to be cancelled. We argue that this is substantially a measurement problem: public benchmarks answer "what can the model do?", whereas a deployment decision requires "is this workflow fit, reliable, safe and worth scaling - here, on our data, under our controls?". We present EnterpriseVal, a use-case-level evaluation system that closes this gap. It comprises (i) a formal specification of the use case and of the frozen socio-technical configuration under test, model, prompts, retrieval, tools, guardrails and human oversight, with an autonomy level and consequence tier that jointly set the required evaluation intensity; (ii) a metric catalogue spanning fidelity, utility, efficiency, reliability, assurance and oversight; (iii) a grading protocol that scales blinded expert judgement with calibrated LLM-as-judge scoring through prediction-powered inference; (iv) a two-tier threshold gate, stated as an executable algorithm, that maps metric vectors with confidence bounds to REJECT/CONDITIONAL/SCALE decisions; and (v) a value-and-risk model in which the reviewer catch rate is a measured parameter. We report a pilot across three workflows in a global bank. In credit-memo drafting, human-graded citation precision reached 88% and hallucination rate 1.6% for the best model against gates of 70% and 5%; in procedure transformation, analyst refinement effort fell from an estimated 27.4 to 2.9 hours per document. We separate established results, documented pilot evidence, the proposed system and open hypotheses, and specify the experiments required for full validation
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 30 of its 65 references that are in the map. First posted on arXiv 18 Sep 2026.
Builds on
30 of 65 references.
References · 30
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2020Measuring Massive Multitask Language Understandingcited by 7,768
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
- 2022Survey of Hallucination in Natural Language Generationcited by 2,056
- 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
- 2024A Survey on LLM-as-a-Judgecited by 1,384
- 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
- 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
- 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 752
- 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 558
18 more
- 2021AI and the Everything in the Whole Wide World Benchmarkcited by 340
- 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 223
- 2023Prediction-Powered Inferencecited by 214
- 2024AI Agents That Mattercited by 199
- 2024Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferencescited by 142
- 2025Measuring AI Ability to Complete Long Software Taskscited by 142
- 2025GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Taskscited by 142
- 2025Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluationcited by 83
- 2026Towards a Science of AI Agent Reliabilitycited by 58
- 2025Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluationcited by 57
- 2025Measuring Agents in Productioncited by 35
- 2025Can LLMs Help You at Work? A Sandbox for Evaluating LLM Agents in Enterprise Environmentscited by 15
- 2025The Measurement Imbalance in Agentic AI Evaluation Undermines Industry Productivity Claimscited by 13
- 2026Log analysis is necessary for credible evaluation of AI agentscited by 8
- 2025Beyond Accuracy: A Multi-Dimensional Framework for Evaluating Enterprise Agentic AI Systemscited by 6
- 2025Enterprise Large Language Model Evaluation Benchmarkcited by 5
- 2025Towards a Standard, Enterprise-Relevant Agentic AI Benchmark: Lessons from 5.5 billion tokens' worth of agentic AI evaluationscited by 3
- 2026Beyond Static Leaderboards: Predictive Validity for the Evaluation of LLM Agentscited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Continuous Process-Level Evaluation for Evolving Enterprise AI Agent Skills
- 2026LH-Bench: Skill-Grounded Evaluation of Long-Horizon Agents on Subjective Enterprise Tasks
- 2026AlphaEval: Evaluating Agents in Production
- 2026BADGER: Bridging Agentic and Deterministic Evaluation for Generative Enterprise Reasoning
- 2026EnterpriseLab: A Full-Stack Platform for developing and deploying agents in Enterprises
- 2026EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings
- 2026StartupBench: Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflows
- 2026Interview-Informed Generative Agents for Product Discovery: A Validation Study