SemVerBench: Benchmarking LLM Comprehension of Version-Constraint Resolution Semantics
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
Large language model (LLM) coding agents constantly decide whether a version satisfies a constraint such as ^1.2.3 or >=2.0,<3, yet their grasp of version-constraint semantics has never been measured directly. We introduce SemVerBench, the first benchmark of LLM version-constraint resolution semantics across three ecosystems (npm, PEP 440, Cargo): 240 machine-checkable items with unique answers, built author-neutrally from four balanced sources (each ecosystem's official test suite plus three frontier LLM proposers) and labeled by a non-circular two-implementation oracle. Evaluating six frontier models, we find systematic, predictable per-mechanism blind spots: a partial-comparator carry rule (>1.2 means >=1.3.0) traps every model on Cargo (near 60%), and although standard PEP 440 prefix matching is universal, on zero-pad/post-release corner cases GPT-5.1 collapses (0/26) while Claude stays at 97-100% (verified on a 67-item oracle-validated set). Opus significantly outperforms all other models, and Sonnet outperforms the OpenAI models (McNemar). The failures look more like an activation/application gap than a knowledge gap: injecting the rule or a light correct hint recovers most errors, whereas interval decomposition does not, and models are at ceiling on the basic forms of the same rules. An author-stratified analysis finds no statistically significant self-favoritism. Because the task is verifiable and a free, 100%-correct resolver exists, tool delegation reaches ~100%: coding agents should delegate version resolution to a resolver rather than reason about versions in-head.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 10 of its 10 references that are in the map. First posted on arXiv 10 Sep 2026.
Builds on
10 of 10 references.
References · 10
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
- 2022Language Models (Mostly) Know What They Knowcited by 1,783
- 2023Towards Understanding Sycophancy in Language Modelscited by 988
- 2024CFBench: A Comprehensive Constraints-Following Benchmark for LLMscited by 65
- 2025DependEval: Benchmarking LLMs for Repository Dependency Understandingcited by 16
- 2025DI-BENCH: Benchmarking Large Language Models on Dependency Inference with Testable Repositories at Scalecited by 7
- 2026ConstraintBench: Benchmarking LLM Constraint Reasoning on Direct Optimizationcited by 6
- 2023I depended on you and you broke me: An empirical study of manifesting breaking changes in client packagescited by 3
- 2024The Ability of Large Language Models to Evaluate Constraint-satisfaction in Agent Responses to Open-ended Requestscited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Constraint Decay: The Fragility of LLM Agents in Backend Code Generation
- 2025The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason
- 2025Beyond Synthetic Benchmarks: Evaluating LLM Performance on Real-World Class-Level Code Generation
- 2026Refine After Generation: Toward Correct and Concise Patches in LLM-based Program Repair
- 2025A Reality Check of Language Models as Formalizers on Constraint Satisfaction Problems
- 2025CoRe: Benchmarking LLMs Code Reasoning Capabilities through Static Analysis Tasks
- 2025LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Research
- 2026CangjieBench: Benchmarking LLMs on a Low-Resource General-Purpose Programming Language