# Continuous Benchmark Generation for Evaluating Enterprise-scale LLM Agents

arXiv: https://arxiv.org/abs/2511.10049 · HTML: https://mlsanity.com/paper/2511.10049/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 4 of its 12 references that are in the map.
- First posted on arXiv 13 Nov 2025.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 4 of 4)

- [STaRK: Benchmarking LLM Retrieval on Textual and Relational Knowledge Bases](https://mlsanity.com/paper/2404.13207/) (2024)
- [LogSage: An LLM-Based Framework for CI/CD Failure Detection and Remediation with Industrial Validation](https://mlsanity.com/paper/2506.03691/) (2025)
- [STACKFEED: Structured Textual Actor-Critic Knowledge Base Editing with FeedBack](https://mlsanity.com/paper/2410.10584/) (2024)
- [Intent-based System Design and Operation](https://mlsanity.com/paper/2502.05984/) (2025)

## Built on by (most cited first, 5 of 5)

- [Anchor: Mitigating Artifact Drift in Agent Benchmark Generation](https://mlsanity.com/paper/2605.26321/) (2026)
- [Running the Gauntlet: Challenging Agentic Tasks](https://mlsanity.com/paper/2606.14397/) (2026)
- [EvalLoop: A Methodology for Evaluation-Driven Iterative Improvement of Business AI Systems](https://mlsanity.com/paper/2607.05638/) (2026)
- [What Could the Agent See at 19:05? Generating Temporal Enterprise Scenarios from Real Research and Replaying Them to Evaluate Agents](https://mlsanity.com/paper/2608.01042/) (2026)
- [VIGIL: Verifier-Informed Gated Improvement Loop for Spreadsheet Question Answering](https://mlsanity.com/paper/2610.04287/) (2026)

## Abstract (arXiv)

The rapid adoption of AI agents across domains has made systematic evaluation crucial for ensuring their usefulness and successful production deployment. Evaluation of AI agents typically involves using a fixed set of benchmarks and computing multiple evaluation metrics for the agent. While sufficient for simple coding tasks, these benchmarks fall short for enterprise-scale agents, where services and requirements evolve continuously and ground-truth examples are sparse. We propose a process of benchmark generation that helps evolve the benchmarks as the requirements change and perform robust evaluation of evolving AI agents. We instantiate this approach for a case study of service migration from one deployment platform to another at a large public enterprise. Our approach relies on semi-structured documents where developers express the high-level intent, and uses state-of-the-art LLMs to generate benchmarks from just a small number of such documents. Overall, this process results in a maintainable evaluation framework, enabling rapid feedback on agent performance and facilitating targeted improvements.