# SemVerBench: Benchmarking LLM Comprehension of Version-Constraint Resolution Semantics

arXiv: https://arxiv.org/abs/2609.11180 · HTML: https://mlsanity.com/paper/2609.11180/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 10 of its 10 references that are in the map.
- First posted on arXiv 10 Sep 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 10 of 10)

- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [CFBench: A Comprehensive Constraints-Following Benchmark for LLMs](https://mlsanity.com/paper/2408.01122/) (2024)
- [DependEval: Benchmarking LLMs for Repository Dependency Understanding](https://mlsanity.com/paper/2503.06689/) (2025)
- [DI-BENCH: Benchmarking Large Language Models on Dependency Inference with Testable Repositories at Scale](https://mlsanity.com/paper/2501.13699/) (2025)
- [ConstraintBench: Benchmarking LLM Constraint Reasoning on Direct Optimization](https://mlsanity.com/paper/2602.22465/) (2026)
- [I depended on you and you broke me: An empirical study of manifesting breaking changes in client packages](https://mlsanity.com/paper/2301.04563/) (2023)
- [The Ability of Large Language Models to Evaluate Constraint-satisfaction in Agent Responses to Open-ended Requests](https://mlsanity.com/paper/2409.14371/) (2024)

## Built on by (most cited first, 1 of 1)

- [Counterfactual Rollout Replay: Forkable Environments as Free Process Rewards for Software Engineering Agents](https://mlsanity.com/paper/2609.33875/) (2026)

## Abstract (arXiv)

Large language model (LLM) coding agents constantly decide whether a version satisfies a constraint such as ^1.2.3 or >=2.0,<3, yet their grasp of version-constraint semantics has never been measured directly. We introduce SemVerBench, the first benchmark of LLM version-constraint resolution semantics across three ecosystems (npm, PEP 440, Cargo): 240 machine-checkable items with unique answers, built author-neutrally from four balanced sources (each ecosystem's official test suite plus three frontier LLM proposers) and labeled by a non-circular two-implementation oracle. Evaluating six frontier models, we find systematic, predictable per-mechanism blind spots: a partial-comparator carry rule (>1.2 means >=1.3.0) traps every model on Cargo (near 60%), and although standard PEP 440 prefix matching is universal, on zero-pad/post-release corner cases GPT-5.1 collapses (0/26) while Claude stays at 97-100% (verified on a 67-item oracle-validated set). Opus significantly outperforms all other models, and Sonnet outperforms the OpenAI models (McNemar). The failures look more like an activation/application gap than a knowledge gap: injecting the rule or a light correct hint recovers most errors, whereas interval decomposition does not, and models are at ceiling on the basic forms of the same rules. An author-stratified analysis finds no statistically significant self-favoritism. Because the task is verifiable and a free, 100%-correct resolver exists, tool delegation reaches ~100%: coding agents should delegate version resolution to a resolver rather than reason about versions in-head.