mlsanityData to 14 Oct 2026

Judges Follow the Reference: The Derivation Gap in Verifying Tool-Call Arguments

arXiv 2610.18497 ↗First posted 13 Oct 2026cs.AI cs.CLcited by 0

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

Tool-call judges evaluate argument values proposed by LLM agents. We find that they accept a correct value less often when their input holds no reference for it, that is, no text that says what the value should be. This happens when the value is a default value from the function schema that the user did not mention, when it has to be inferred, or when the text it comes from was cut from the judge input. We call this drop in acceptance the derivation gap. It appears across all seven judges we evaluate, including System One models CLM and Jev. The gap is not only a truncation artifact: correct default values have the largest gap on a benchmark with no truncation. A controlled source-relocation test shows that judges follow the reference in their input: they favor a visible incorrect value over a correct value whose source is hidden. We introduce DeRef (Derived Reference): a separate model reads the full history without seeing the candidate and writes a reference for the judge. Across three benchmarks, DeRef improves overall argument AUROC for all judges tested. On the controlled test, it raises AUROC from chance or below to between 98.40 and 99.59.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 32 of its 49 references that are in the map. First posted on arXiv 13 Oct 2026.

Builds on

32 of 49 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026TwinCheck: Evidence-Grounded Negative-Twin Verification for Stateful Tool Agents22 Sep
  2. 2026AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling27 Aug
  3. 2026AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents15 Mar
  4. 2026ToolPRMBench: Evaluating and Advancing Process Reward Models for Tool-using Agents18 Jan
  5. 2025ToolPRM: Fine-Grained Inference Scaling of Structured Outputs for Function Calling16 Oct
  6. 2025ToolRM: Outcome Reward Models for Tool-Calling Large Language Models15 Sep
  7. 2025ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling under Long-Context Scenario17 Jan

Other references · 25

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2023Let's Verify Step by Stepcited by 3,789
  5. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  6. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 415
  7. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 302
  8. 2021Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarizationcited by 129
  9. 2025NoLiMa: Long-Context Evaluation Beyond Literal Matchingcited by 126
  10. 2022Evaluating the Factual Consistency of Large Language Models Through News Summarizationcited by 92
  11. 2024Sufficient Context: A New Lens on Retrieval Augmented Generation Systemscited by 72
  12. 2025No Free Labels: Limitations of LLM-as-a-Judge Without Human Groundingcited by 54
13 more
  1. 2024Molecular Facts: Desiderata for Decontextualization in LLM Fact Verificationcited by 44
  2. 2024Decomposition Dilemmas: Does Claim Decomposition Boost or Burden Fact-Checking Performance?cited by 36
  3. 2026JEV-as-a-Judge: Accept When Confident, Escalate When Unsurecited by 30
  4. 2026Agentic Rubrics as Contextual Verifiers for SWE Agentscited by 25
  5. 2024RevisEval: Improving LLM-as-a-Judge via Response-Adapted Referencescited by 20
  6. 2026Agent-Sentry: Bounding LLM Agents via Execution Provenancecited by 15
  7. 2024Do Automatic Factuality Metrics Measure Factuality? A Critical Evaluationcited by 11
  8. 2026The Granularity Mismatch in Agent Security: Argument-Level Provenance Solves Enforcement and Isolates the LLM Reasoning Bottleneckcited by 11
  9. 2026More Convincing, Not More Correct: Self-Play Reward Hacking of Reference-Free LLM Judgescited by 10
  10. 2026Safeguarding LLM Agents from Misalignment through Provenance Analysiscited by 6
  11. 2026Judging Against the Reference: Uncovering Knowledge-Driven Failures in LLM-Judges on QA Evaluationcited by 5
  12. 2026FinTrace: Holistic Trajectory-Level Evaluation of LLM Tool Calling for Long-Horizon Financial Taskscited by 4
  13. 2026LLM Judges Can Be Too Generous When There Is No Reference Answercited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026AgentLTL: A Trace-Verification Framework for Measuring, Enforcing, and Training Procedural Compliance in Tool-Using LLM Agents
  2. 2026Judging Is Not Enumerating: Silent Omissions in LLM-Authored Acceptable Sets
  3. 2026Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guarantees
  4. 2026LLM Agents Already Know When to Call Tools -- Even Without Reasoning
  5. 2026Rethinking Atomic Decomposition for LLM Judges: A Prompt-Controlled Study of Reference-Grounded QA Evaluation
  6. 2026ToolRobustBench: Stage-Wise Perturbation Evaluation and Failure Diagnosis for Tool-Calling Agents
  7. 2026Bias in the Loop: Auditing LLM-as-a-Judge for Software Engineering
  8. 2026ProofJudge: Tool-Grounded LLM Evaluation of Formal Proof Quality in Mathlib