Efficient Benchmarking in Production: A Study of an Evolving LLM Agent
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
Production LLM agents are evaluated repeatedly as they evolve, but full agent benchmarks are costly to rerun. We study efficient recurring evaluation for a production analytics agent serving tens of thousands of monthly active users and report first-hand deployment experience. Using 574 historical runs of the production benchmark, split chronologically into calibration and held-out periods, we compare random sampling, historical caching, fixed representative subsets, and IRT-based adaptive testing. The results show that multidimensional 2PL adaptive testing achieves the best overall score fidelity: executing 200 questions, 38.5% of a full run, yields 1.03 pp of MAE. We nevertheless deployed difficulty-stratified fixed subsets because of their operational simplicity, and show they transfer without recalibration to five other agent families and remain stable across calibration windows as short as one day. Drawing on this deployment experience, we report practical recommendations for recurring production-agent evaluation.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 10 of its 22 references that are in the map. First posted on arXiv 18 Sep 2026.
Builds on
10 of 22 references.
References · 10
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
- 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
- 2023GAIA: a benchmark for General AI Assistantscited by 1,084
- 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 295
- 2023Anchor Points: Benchmarking Models with Much Fewer Examplescited by 84
- 2025Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluationcited by 83
- 2024Active Evaluation Acquisition for Efficient LLM Benchmarkingcited by 33
- 2025Reliable and Efficient Amortized Model-based Evaluationcited by 29
- 2018Predictive Test Selectioncited by 8
- 2024Precise Model Benchmarking with Only a Few Observationscited by 5
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Beyond Static Leaderboards: Predictive Validity for the Evaluation of LLM Agents
- 2025Continuous Benchmark Generation for Evaluating Enterprise-scale LLM Agents
- 2026Benchmarks are Not Enough: RAMP for Runtime Assessing of Agentic Models in Production Systems
- 2026DBA-Bench: A Production-Fidelity Benchmark for LLM-Based Database Operations Agents
- 2026"LLM Agent Performance" Is Not a Single Evaluation Target
- 2026AlphaEval: Evaluating Agents in Production
- 2026ReliabilityBench: Evaluating LLM Agent Reliability Under Production-Like Stress Conditions
- 2026RSIBench-Data: Benchmarking Data-Centric Research for Recursive Self-Improvement