mlsanityData to 7 Oct 2026

Noise Floor Audit for Agent Benchmarks

arXiv 2608.22331 ↗First posted 23 Aug 2026cs.CLcited by 0

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

We audit measurement variability for 3 native tool-calling endpoints across 2 providers on the official BFCL multiple and parallel categories, using matched AST grading. At temperature 0, reruns are nearly deterministic across Groq endpoints and a thinking-enabled Gemini setting: ever-flip fractions are 0.7%, 2.0%, and 2.7%, with mean run correlations of 0.997, 0.966, and 0.961. Semantics-preserving prompt perturbations create the larger floor on all endpoints, with median perturbation paired SDs 11x to 58x larger than rerun paired SDs. The failure character also shifts: malformed-output failures account for 30%, 7%, and <1% of task failures, so marginal accuracy hides not only stability but also failure mode.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 29 of its 34 references that are in the map. First posted on arXiv 23 Aug 2026.

Builds on

29 of 34 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024ToolScan: A Benchmark for Characterizing Errors in Tool-Use LLMs20 Nov
  2. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs31 Jul
  3. 2023Gorilla: Large Language Model Connected with Massive APIs24 May

Other references · 26

  1. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  2. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  3. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 299
  4. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 218
  5. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
  6. 2019Show Your Work: Improved Reporting of Experimental Resultscited by 164
  7. 2024Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluationscited by 130
  8. 2024OLMES: A Standard for Language Model Evaluationscited by 100
  9. 2023T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Stepcited by 91
  10. 2024What Did I Do Wrong? Quantifying LLMs' Sensitivity and Consistency to Prompt Engineeringcited by 88
  11. 2026From Agent Traces to Trust: A Survey of Evidence Tracing and Execution Provenance in LLM Agentscited by 29
  12. 2025Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluationcited by 22
14 more
  1. 2025Learning Human-Perceived Fakeness in AI-Generated Videos via Multimodal LLMscited by 16
  2. 2026SafetyRepro: Configuration-Conditional Rank Instability on Alignment Benchmarkscited by 14
  3. 2026MemTX: Transactional Belief Commit for Stateful Agent Memorycited by 12
  4. 2026Pre-Registering the Detectable Effect: A Paired-MDE Budget for 4-bit Quantization Benchmarks, with a Pilot Auditcited by 8
  5. 2026Beyond Accuracy: Measuring Bias Acknowledgment in Chain-of-Thought Reasoning for Responsible AI Evaluationcited by 8
  6. 2026How Early Is Early Enough? Design-Dependent Observation-Window Sufficiency in Subscription Churn Predictioncited by 7
  7. 2026MAP-Graph: Provenance-Aware Shared Memory for Multi-Agent Workflowscited by 7
  8. 2026From Reaction to Anticipation: Proactive Failure Recovery through Agentic Task Graph for Robotic Manipulationcited by 6
  9. 2026Do Time Series Foundation Model Benchmarks Hide Regime-Dependent Failures? Evidence from Traffic Speed Forecastingcited by 5
  10. 2026From Faulty Memories to Corrected Actions: Dependency-Guided Rollback Repair for Memory-Augmented Agentscited by 4
  11. 2026Chain-Aware Encoding for Microservice Trace Anomaly Detectioncited by 3
  12. 2026PerfAgent: Profiler-Guided Iterative Refinement for Repository-Level Code Optimizationcited by 2
  13. 2026Same Question, Different Answer? Measuring and Mitigating Prompt Privilege for Equitable AI Accesscited by 2
  14. 2026SphUnc: Hyperspherical Uncertainty Decomposition and Causal Identification via Information Geometrycited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Benchmarking the Benchmarks: A Validity Audit of Tool-Calling Evaluation
  2. 2026The Double Measurement Confound in Agent Benchmarks: De-Scaffolding, Ground-Truth Scoring, and Reliability Beyond the Mean
  3. 2026Beyond Function Calling: Benchmarking Tool-Using Agents under Tool-Environment Unreliability
  4. 2026Loud Failures, Quiet Failures: Fault Detection and Recovery in Tool-Using Language Model Agents
  5. 2026ReliabilityBench: Evaluating LLM Agent Reliability Under Production-Like Stress Conditions
  6. 2026Flat Score, Amplified Failures: How the Error Budget Masks Damage in Quantized LLM Agents
  7. 2026Calibration is the Bottleneck: An Action-Class Diagnostic of Multi-Turn Tool-Calling
  8. 2026When Simulation Lies: A Sim-to-Real Benchmark and Domain-Randomized RL Recipe for Tool-Use Agents