# When Agent Metrics Measure Different Things: An Evidence-Grounded Audit of the Praxa AI Pipeline

arXiv: https://arxiv.org/abs/2609.12017 · HTML: https://mlsanity.com/paper/2609.12017/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 10 references that are in the map.
- First posted on arXiv 10 Sep 2026.

## Builds on (most cited first, 8 of 8)

- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [AI Agents That Matter](https://mlsanity.com/paper/2407.01502/) (2024)
- [AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories](https://mlsanity.com/paper/2504.08942/) (2025)
- [Heterogeneous Agent Collaborative Reinforcement Learning](https://mlsanity.com/paper/2603.02604/) (2026)
- [HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?](https://mlsanity.com/paper/2609.01437/) (2026)
- [Multimodal Duplex Interaction Agent](https://mlsanity.com/paper/2609.08977/) (2026)
- [Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills](https://mlsanity.com/paper/2609.02749/) (2026)
- [AutoResearch: Insight In, Hallucination Out](https://mlsanity.com/paper/2608.17906/) (2026)

## Abstract (arXiv)

Agent evaluations can be numerically correct while measuring a different construct from the one implied by their labels. We present a retrospective measurement audit of selected Praxa AI implementation files, historical evaluation artifacts, and operational records. A 139-case offline routing report contains 112 passes and 27 failures despite zero gating failures, because known gaps are explicitly exempted from the gate. An identifier-free export represents 8,843 tool-attempt rows: 8,395 recorded durations and 448 missing values. Of the durations, 121 equal the signed 32-bit maximum and carry abandoned-client labels; inspected database code clamps elapsed lifecycle age. The pooled recorded 99th percentile is 2,147,483,647 ms, versus 38,118.31 ms among server-observed completed calls. This is a stratum contrast, not a treatment effect. In a documented single-trajectory compaction pilot, the reported follow-up input reduction is 94.39%, but the reduction across the trigger and follow-up calls together is 46.54%. We reproduce the descriptive calculations, verify 91 timing statistics through a separate weighted rational-arithmetic implementation, and execute 13 scoring-function and 12 analysis-verifier tests. Finite-completion bounds show how missing durations limit all-row timing statements without imputing values. The contribution is a source-linked case study and reusable verification package for separating gate policy, lifecycle timing, and request-level accounting from broader agent-performance claims. Historical provider runs and the full current pipeline were not independently reproduced; general capability superiority and population-level statistical significance are not established.