Does a Language Server Save Tokens for Coding Agents? A Measurement Methodology and Preliminary Study
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
Coding agents spend most of their context budget on retrieval. Lexical retrieval (grep) is universal, instant, and zero-setup, but noisy: it cannot tell a definition from a call from a comment. Semantic retrieval via the Language Server Protocol (LSP) is precise and typed, but needs a running, indexed server and pays a per-symbol round-trip. The claim that semantic retrieval is more token-efficient is, we find, asserted almost everywhere and measured almost nowhere: no public source isolates the LSP-vs-lexical token delta for an agent at equal task-success. This paper formalizes the question with one metric (tokens-to-success), specifies a five-arm ablation isolating semantic retrieval from confounds, maps three pre-stated failure modes onto measurable variables, and reports a preliminary study (Python and TypeScript repos; Claude Opus 4.8, Sonnet 4.6, Haiku 4.5). The answer is conditional and usually negative. On symbol-named localization the LSP costs tokens (+6% to +118%) and the agent ignores it when free. On reference-completeness it buys precision but not token savings and cannot raise the recall ceiling set by agent thoroughness; it saves tokens only for the weakest model. Tool choice is task-dependent: models default to grep on localization (0-6% semantic use) but reach for the LSP about half the time on reference tasks, unprompted. On edits scored by real test execution the gap is starkest: grep solves multi-file renames perfectly, a location-only LSP fails three-quarters of them by missing a call site, and even a complete, index-warmed, text-enriched LSP (each reference's line inline, as production LSP-MCP servers do) recovers most of the gap but cannot close it, since a rename must touch comments and strings that semantic references exclude. The implication is not LSP-always but an adaptive router keyed on task class, model capability, and lexical noise.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 6 of its 11 references that are in the map. First posted on arXiv 29 Jun 2026.
Builds on
6 of 11 references.
Closest ancestors · 3
Same direction, newest first
- 2026Code as Agent Harness18 May
- 2025Reinforcement Learning from Compiler and Language Server Feedback27 Oct
- 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering6 May
Other references · 3
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
- 2024AutoCodeRover: Autonomous Program Improvementcited by 278
- 2026TypeScript Repository Indexing for Code Agent Retrievalcited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026The Best Programming Language for Tokenmaxxing: An Investigation of Coding Agent Behavior Across Programming Languages
- 2026Agent Retrieval Bench: Evaluating Repository Context Retrieval for Coding Agents
- 2026Is Grep All You Need? How Agent Harnesses Reshape Agentic Search
- 2026Token-Native Storage: Read and Write in your Agent's Language
- 2026Cross-Lingual Token Arbitrage: Optimizing Code Agent Context Windows via Local LLM Preprocessing
- 2026Notation Matters: A Benchmark Study of Token-Optimized Formats in Agentic AI Systems
- 2026LARGER: Lexically Anchored Repository Graph Exploration and Retrieval
- 2026Agents Don't Paginate: First-Chunk Selection for LLM Tool Responses