Interface-Induced Trajectory Censoring
Not on the map. Why?
Abstract (arXiv)
Agent evaluations report a tool-call rate read off the serving stack. That number can be zero while the model is emitting well-formed calls: the interface censors the trajectory before anything downstream sees it. On BFCL v4's own data, executor and scorer, holding weights, cases, decoding and seeds fixed and changing only the serving adapter, the same model scores 0.00 or 0.96 / 0.19. A 2x2 over chat template and parser locates the effect exactly: both main effects are exactly zero and all of it sits in the interaction -- no component is defective, and repairing one side of the contract buys precisely nothing. On tau-bench's 115 interactive retail tasks the same swap moves server-parsed calls from 0 to 636 and tasks reaching any tool execution from 0 to 103. Our probe reproduces the funnel across a 21x scale range of Qwen2.5-Coder: the server parses 0/100 at every size while well-formed emitted calls rise to 80/100 at 32B (~72 after calibration against an adjudicated gold standard). Under a matched envelope, across a comparable scale span, the silent fraction stays at 0-2, a prediction committed to the repository before the run. Llama-3.1-8B's 23% rate of calling the task function itself as a tool falls to 0 under one strict:true flag. The mismatch reaches inside the training loop, and its consequence is scale-dependent: in verl's AgentLoop at 7B, 45 of 115 generations carry a complete call; 0 are accepted, 0 execute, 0 return an observation. At 1.5B the same zero is over-determined, so we report the two scales separately. At evaluation time, repairing the adapter restores the mechanism but not a significant outcome gain: parsing 0->84, rescues 0->9, pass rate 53->62 (n.s.). We release a 98-line preflight check that catches every silent failure here. The observed tool-call rate is not a property of the model alone; it is a property of the model-interface stack that measures it.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 18 of its 27 references that are in the map. First posted on arXiv 3 Sep 2026.
Builds on
18 of 27 references.
References · 18
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
- 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
- 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
- 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
- 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
- 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
- 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 627
- 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 591
- 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 310
6 more
- 2021The Benchmark Lotterycited by 136
- 2026ToolFailBench: Diagnosing Tool-Use Failures in LLM Agentscited by 11
- 2026Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failurescited by 10
- 2026Benchmarking the Benchmarks: A Validity Audit of Tool-Calling Evaluationcited by 8
- 2026When the Ruler is Broken: Parsing-Induced Suppression in LLM-Based Security Log Evaluationcited by 3
- 2026Beyond the Leaderboard: A Synthesis of Tool-Use, Planning, and Reasoning Failures in Large Language Model Agentscited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.