When the Tool Decides: LLM Agents Defer Blindly to Graph Neural Network Tools, and Stronger Backbones Defer More
On the map: LLM agents and tool use → Web, search, and research agents
Abstract (arXiv)
A growing line of work equips large language model (LLM) agents with graph neural networks (GNNs) as callable tools, assuming the agent exercises judgment over when and how much to rely on such a tool. We test this directly. We expose a frozen GNN to a ReAct-style LLM agent as an explicit tool and measure, on node classification over a text-attributed graph (ogbn-arxiv, replicated on WikiCS), whether the agent uses the tool or merely obeys it. We find the agent does not exercise judgment: its predictions agree with the raw GNN's 97.6-99.2% of the time (5 seeds), collapsing into a GNN parrot that adopts the tool's output wholesale and bypasses its own reasoning. Sweeping backbone capability (Qwen2.5 0.5B-7B), the deference is not a weak-model artifact: among models able to invoke the tool, agreement rises with capability (0.60 to 0.98 from 1.5B to 7B). Crucially, the cost of deference does not shrink as capability grows and grows where alternatives emerge: a per-node oracle over the available actions beats the parrot by 0.09-0.18 at 3B and 0.12-0.22 at 7B, roughly doubling at high homophily, because the parrot is pinned to the frozen GNN while the agent's alternatives improve; at 7B a simple neighbour-label tool overtakes the GNN at high homophily (0.81 vs 0.71) yet the agent still defers. A simple selective-invocation gate recovers about half of that high-homophily gap (0.71 to 0.83) but yields no net global gain, and held-out estimates bound the best achievable gate over standard test-time features to at most a third of the oracle headroom: reliable selective invocation looks limited by available information, not merely router design. Our results are a cautionary measurement: evaluations of agent+tool systems cannot assume the agent adds judgment on top of the tool, and selective invocation must be designed in rather than expected to emerge from scale.
Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 9 of its 10 references that are in the map. First posted on arXiv 12 Jun 2026.
Builds on
9 of 10 references.
Closest ancestors · 1
Same direction, newest first
Other references · 8
- 2026Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgetscited by 30
- 2025Position: Agent Should Invoke External Tools ONLY When Epistemically Necessarycited by 22
- 2025The Reasoning Trap: How Enhancing LLM Reasoning Amplifies Tool Hallucinationcited by 14
- 2026Are Tools All We Need? Unveiling the Tool-Use Tax in LLM Agentscited by 11
- 2026Investigating Tool-Memory Conflicts in Tool-Augmented LLMscited by 6
- 2026GNN-as-Judge: Unleashing the Power of LLMs for Graph Learning with GNN Feedbackcited by 4
- 2025When Structure Doesn't Help: LLMs Do Not Read Text-Attributed Graphs as Effectively as We Expectedcited by 3
- 2025Glance for Context: Learning When to Leverage LLMs for Node-Aware GNN-LLM Fusioncited by 2
Built on by
Citing papers per year: 2026: 6.
6 papers.
- 2026MemToC: Benchmarking Memory-Tool Conflict Resolution in Large Language Modelscited by 1
- 2026Agents Trust Tools Too Much: Measuring Reliance on Unreliable Toolscited by 1
- 2026What Does a Skill Actually Do? Estimands and Evaluation Validity for Tool and Skill Use in LLM Agents: A Critical Reviewcited by 1
- 2026The Civilization Framework: Sovereign-Anchored Communication Between Personal Multi-Agent Systemscited by 0
- 2026What Can Component-Replacement Evidence Establish? A Critical Scoping Review of Local Decisions in LLM Agentscited by 0
- 2026RSure-Agent: Reliable Use of Tool Observations for Remote Sensing Agentscited by 0
- 2026RSure-Agent: Reliable Use of Tool Observations for Remote Sensing Agents4 Oct 2026
- 2026What Can Component-Replacement Evidence Establish? A Critical Scoping Review of Local Decisions in LLM Agents30 Sep 2026
- 2026What Does a Skill Actually Do? Estimands and Evaluation Validity for Tool and Skill Use in LLM Agents: A Critical Review27 Sep 2026
- 2026Agents Trust Tools Too Much: Measuring Reliance on Unreliable Tools4 Sep 2026
- 2026The Civilization Framework: Sovereign-Anchored Communication Between Personal Multi-Agent Systems3 Sep 2026
- 2026MemToC: Benchmarking Memory-Tool Conflict Resolution in Large Language Models26 Aug 2026
Similar, not cited
Close in wording, no citation either way
- 2025A Graph Talks, But Who's Listening? Rethinking Evaluations for Graph-Language Models
- 2024Can Graph Learning Improve Planning in LLM-based Agents?
- 2026Are Large Language Models Suitable for Graph Computation? Progress and Prospects
- 2026GTA: Graph Theory Agent and Benchmark for Algorithmic Graph Reasoning with LLMs
- 2024Can LLM Graph Reasoning Generalize beyond Pattern Memorization?
- 2025GraphMaster: Automated Graph Synthesis via LLM Agents in Data-Limited Environments
- 2025Automated Decision-Making on Networks with LLMs through Knowledge-Guided Evolution
- 2025GraphOmni: A Comprehensive and Extensible Benchmark Framework for Large Language Models on Graph-theoretic Tasks