mlsanityData to 14 Oct 2026

When Agent Evaluation Becomes Unreliable: An Empirical Study of Oracle Failures and Configuration-Dependent Measurement in an LLM Agent Benchmark

arXiv 2610.16067 ↗First posted 12 Oct 2026cs.SE cs.AIcited by 0

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Agent evaluation harnesses are software systems that run test suites, generate inputs, and decide outcomes with test oracles. This paper audits such software for reliability defects that score tables cannot reveal. We introduce an oracle-integrity auditing approach with five state-integrity invariants and three known-answer controls. We evaluate it by injecting seven oracle defect classes across two task domains and a synthetic domain. All injected defect classes are detected, and the repaired oracle has no false positives. Applied to an agent benchmark, the audit reveals a reward-function defect that changes the environment state before post-run scoring. Across 4,320 episodes, replay-based correction changes database-pass labels from 4,194 to 2,883 and reduces measured success from 94.3% to 65.5%. We also study configuration-dependent measurement effects. A component ablation over 3,240 episodes shows that the agent's policy documents, tool descriptions, and tool error strings influence observed trajectory reliability, with tool descriptions showing the largest exploratory effect (-9.2 pp) and error strings the smallest (-5.3 pp). A second task domain reverses the sign of the language effect, so the result is bounded to the tested configuration. Finally, we report a portability boundary: the auditing approach cannot validate a second independently implemented benchmark because the required reference terminal state cannot be reconstructed from its recorded API-call log. The paper releases a replication package containing the auditing suite, fault-injection catalogue, frozen records, deviation logs, and offline analysis entry points without network access, API keys, or model calls.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 27 of its 39 references that are in the map. First posted on arXiv 12 Oct 2026.

Builds on

27 of 39 references.

Closest ancestors · 1

Same direction, newest first

  1. 2026Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents31 Mar

Other references · 26

  1. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  2. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 627
  3. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 399
  4. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 367
  5. 2019Machine Learning Testing: Survey, Landscapes and Horizonscited by 218
  6. 2025Survey on Evaluation of LLM-based Agentscited by 204
  7. 2026Mind the Sim2Real Gap in User Simulation for Agentic Taskscited by 54
  8. 2025SimulatorArena: Are User Simulators Reliable Proxies for Multi-Turn Evaluation of AI Assistants?cited by 50
  9. 2026Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluationscited by 40
  10. 2024Reliable LLM-based User Simulator for Task-Oriented Dialogue Systemscited by 25
  11. 2026A Survey on LLM-based Conversational User Simulationcited by 18
  12. 2026Assessing the Reliability of Persona-Conditioned LLMs as Synthetic Survey Respondentscited by 10
14 more
  1. 2026Different Demographic Cues Yield Inconsistent Conclusions About LLM Personalization and Biascited by 4
  2. 2026Dialect vs Demographics: Quantifying LLM Bias from Implicit Linguistic Signals vs. Explicit User Profilescited by 4
  3. 2026Multilingual Prompt Localization for Agent-as-a-Judge: Language and Backbone Sensitivity in Requirement-Level Evaluationcited by 2
  4. 2026Verifiable User Simulation for Search and Recommendation Systemscited by 2
  5. 2026Rank Confidence Sequences:Anytime-valid Leaderboardscited by 2
  6. 2026Do Agent Benchmarks Do What They Say? An Executable-Contract Audit of Tool-Using Agent Environmentscited by 2
  7. 2026Does the Judge Prefer English? Evaluating Language-Switching Invariance in LLM-as-a-Judgecited by 1
  8. 2026PolyWorkBench: Benchmarking LLM Agents for Cross-Lingual Long-Horizon Workflowscited by 1
  9. 2026KnowSim: Evaluating Information Calibration in LLM Assistants with User Simulators that Learncited by 1
  10. 2026Are Near-Tied LLM Rankings Robust to Family-DIF-Guided Benchmark Recomposition?cited by 1
  11. 2026Disclosure-Gated User Simulation for Companion-Agent Evaluationcited by 1
  12. 2026Translating the Translator: Decomposing the Cost of English-Forced Inter-Agent Communicationcited by 1
  13. 2026BabelArena: A Large-Scale Multilingual Benchmark for LLM Agentscited by 1
  14. 2026UserProxyBench: Evaluating LLM User Simulators for Agent Benchmarks and Trainingcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Why Static Skill Inconsistencies Do Not Equate to Runtime Risks: A Behavioral Analysis of LLM Agent Execution
  2. 2026Beyond Task Completion: Revealing Corrupt Success in LLM Agents through Procedure-Aware Evaluation
  3. 2026No Task Fails Every Time: Why One-Shot Audits Are Structurally Blind to Agent Damage
  4. 2026AgentFixer: From Failure Detection to Fix Recommendations in LLM Agentic Systems
  5. 2026Designing Reliable LLM-as-a-Judge Measurement Systems for Multi-Turn Business Agents
  6. 2026DBA-Bench: A Production-Fidelity Benchmark for LLM-Based Database Operations Agents
  7. 2026BenchShield: Formal Model-Backed Instrumentation for Reward Integrity in LLM-Agent Evaluation Infrastructure
  8. 2026AgentAtlas: Beyond Outcome Leaderboards for LLM Agents