mlsanityData to 7 Oct 2026

Layer-Isolated Evaluation: Gating the Deterministic Scaffold of a Production LLM Agent with a No-LLM, Regression-Locked Test Harness

arXiv 2606.11686 ↗First posted 10 Jun 2026cs.CL cs.AIcited by 0

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

End-to-end task-success is the dominant way to evaluate LLM agents, but one aggregate number tells you that an agent regressed, not where. We present layer-isolated evaluation: a deployed ordering agent is decomposed into a fixed taxonomy of layers (ontology, intent, routing, decomposition, escalation, safety, memory, and cross-cutting envelope/defense), each exercised by its own assertion slice in a deterministic, no-LLM "pure" mode. The pure suite (238 cases across 23 slices; 225 run in 2.39 s, ~10 ms/case) runs in CI on every change against a locked per-slice baseline. We validate by controlled regression injection, degrading one layer at a time across seven non-safety layers. The effect we did not design in is masking: the aggregate pass-rate barely moves (-1.7 to -5.9 pp for six local regressions), while the matching slice craters (-25 to -91 pp). A layer's slice reacting to its own fault is partly by construction; the measured results are (i) the aggregate masking and (ii) that damage stays off the other slices: the injected layer's slice is the single worst-hit in 5 of 7 cases and top-3 in 7 of 7 (mean rank 1.29 of 19). Localization replicates on a second, structurally different tenant (Starbucks SG): all seven matching slices crater, so it is not a single-catalog artifact. We position it as a concrete, deterministic instantiation of the component-level evaluation EDDOps prescribes but leaves unimplemented, with CheckList as ancestor and as the deterministic mirror image of whole-workflow stochastic mutation testing. Our contributions: (a) a fully decomposed, sub-second, no-LLM per-layer harness for a production agent, (b) a coverage-honesty test-adequacy criterion that refuses to score an unexercised layer, and (c) the regression-injection demonstration that per-slice baseline-locked gates localize regressions an aggregate metric masks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 17 of its 18 references that are in the map. First posted on arXiv 10 Jun 2026.

Builds on

17 of 18 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026Catching One in Five: LLM-as-Judge Blind Spots in Production Multi-Turn Transaction Agents9 Jun
  2. 2026Agent Planning Benchmark: A Diagnostic Framework for Planning Capabilities in LLM Agents3 Jun
  3. 2026AgentAtlas: Beyond Outcome Leaderboards for LLM Agents19 May
  4. 2025Evaluation and Benchmarking of LLM Agents: A Survey29 Jul
  5. 2024Evaluation-Driven Development and Operations of LLM Agents: A Process Model and Reference Architecture21 Nov
  6. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains17 Jun
  7. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents24 Jan
  8. 2023AgentBench: Evaluating LLMs as Agents7 Aug

Other references · 9

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  4. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  5. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  6. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  7. 2019Machine Learning Testing: Survey, Landscapes and Horizonscited by 213
  8. 2024AgentFL: Scaling LLM-based Fault Localization to Project-Level Contextcited by 47
  9. 2026AgentAssay: Token-Efficient Regression Testing for Non-Deterministic AI Agent Workflowscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Evaluate the Stack, Not the Layer: Do Deterministic and LLM Gates for Agent Actions Fail Independently?
  2. 2026GLIDE: Generalized Layer-wise Intrinsic Distributional Evaluation for Heterogeneous LLM Agents
  3. 2026AgentHPOBench: A Benchmark For Evaluating LLM Agents as Sequential Hyperparameter Optimizers
  4. 2026Efficient Benchmarking in Production: A Study of an Evolving LLM Agent
  5. 2026PACE: A Proxy for Agentic Capability Evaluation
  6. 2026Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents
  7. 2026Verification as an Architectural Layer for LLM Agents: A V-Model Design, and a Pilot Study of Its Deterministic Core
  8. 2026TraceCompiler: Skill-Guided Mining and Compilation of LLM Agent Traces into Mostly Deterministic Workflows