When Are Aggregate Agent Traces Diagnosable? Traffic-Governed Interpretation and Calibrated Abstention
Not on the map. Why?
Abstract (arXiv)
Runtime traces can appear transparent, but a closed-loop policy determines which states are visited and which failures become visible. We study a simulated hotel-pricing agent mapping time, inventory, and market state to discrete price actions under varying demand regimes. A fault may leave no aggregate trace when the policy rarely visits affected cells. We treat entry into aggregate-only fault interpretation as a diagnosability decision preceding scoring or localization. A reference-map gate requires repeated clean-policy support; a matched runtime gate then requires joint support in clean and current streams. Signal analysis occurs only after both pass. We calibrate false admission on a disjoint clean stream at the physical-component level and model detection by affected clean traffic rather than nominal cell coverage. In a frozen one-shot heldout, 55/72 (76.4%) regime-component units were reference-admitted, representing 20 physical components; 54/55 passed matched runtime admission, while the rejected unit abstained. Stable false admission was 0/20, with a one-sided exact 95% upper bound of 0.1391, meeting the frozen 0.20 criterion. Across 540 repeated unit-arm rows nested in those 20 clusters, affected clean traffic reduced negative log likelihood by 29.3% relative to cell coverage, a gain of 0.1264 nats per row (cluster-bootstrap 95% interval [0.0593, 0.1918]). Adding mask family and its interaction improved log loss by 0.0015 nats per row (one-sided upper bound 0.0066), below the frozen 0.01 practical-sufficiency margin. A development audit found that exact minimum hitting set and greedy selection chose identical supports in 12/12 scenarios because singleton evidence had resolved the conflicts. The result is a bounded rule for interpreting aggregate agent behavior: first establish exposure, then score change, and abstain when the trace cannot support the claim.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 12 of its 23 references that are in the map. First posted on arXiv 22 Sep 2026.
Builds on
12 of 23 references.
References · 12
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
- 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
- 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
- 2016Distribution-Free Predictive Inference For Regressioncited by 668
- 2020Is Pessimism Provably Efficient for Offline RL?cited by 364
- 2019SelectiveNet: A Deep Neural Network with an Integrated Reject Optioncited by 350
- 2021Distribution-Free, Risk-Controlling Prediction Setscited by 288
- 2018Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shiftcited by 243
- 2020Selective Classification via One-Sided Predictioncited by 38
- 2019Characterization of Overlap in Observational Studiescited by 15
- 2024Selective Classification Under Distribution Shiftscited by 12
- 2025The Limits of Predicting Agents from Behaviourcited by 3
Built on by
Not cited yet.