Measuring What a Specification Determines: A Formal Semantic-Block Model and an Execution-Judged Benchmark
Not on the map. Why?
Abstract (arXiv)
This work introduces a formal semantic-block model for specifications and an execution-judged benchmark for evaluating specification quality independently of model capability. A specification is represented as a structure comprising semantic blocks, dependency relations, block-owned rules, decision points, and explicitly open questions, subject to four machine-checkable well-formedness conditions: acyclicity, single ownership, constraint domination, and totality or ambiguity-stop. Determinacy is defined model-theoretically as agreement among all conforming implementations and is estimated empirically through convergence across independent implementers. The model is instantiated on an Oracle-to-PostgreSQL migration specification containing 18 blocks and 19 dependency edges. Computational validation shows that the five-layer decomposition reduces mean per-task context by approximately 71% through dependency closures, covers 85.5% of the study-defined Oracle construct taxonomy with all identified gaps triaged, is not Pareto-dominated by the tested alternative partitions, and is recovered at the 99.9th percentile from citation-derived edges not used to define the original structure. The benchmark keeps the implementer panel fixed, includes a mandatory no-specification control arm, and uses PostgreSQL 16 and a live Oracle instance as deterministic execution judges. Six designed studies, including three pre-registered manipulations and three diagnostic analyses, further examine specification effects. Repeated runs on a 25-unit subsample reveal an empirical variability floor with a median arm-delta spread of 14.4 percentage points. The results support determinacy as a formal concept but not as a standalone empirical quality metric for the evaluated contemporary LLM implementers.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 20 of its 37 references that are in the map. First posted on arXiv 19 Aug 2026.
Builds on
20 of 37 references.
References · 20
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
- 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
- 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
- 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 417
- 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
- 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
- 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 223
- 2023An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generationcited by 193
- 2022Do Users Write More Insecure Code with AI Assistants?cited by 186
- 2023Large Language Models for Software Engineering: Survey and Open Problemscited by 178
- 2020Semantic Evaluation for Text-to-SQL with Distilled Test Suitescited by 123
- 2024On the Evaluation of Large Language Models in Unit Test Generationcited by 33
8 more
- 2026Spec-Driven Development:From Code to Contract in the Age of AI Coding Assistantscited by 14
- 2016Rapid quality assurance with Requirements Smellscited by 11
- 2026Specification-Based Code-Text-Code Reengineering for LLM-Mediated Software Evolutioncited by 5
- 2026Unified Architecture Metamodel of Information Systems Developed by Generative AIcited by 4
- 2026Token Optimization Strategies for LLM-Based Oracle-to-PostgreSQL Migrationcited by 3
- 2025Fine-tuned LLM-based Code Migration Frameworkcited by 2
- 2026Specification-Driven Development Benchmark: Security Knowledge Transitioncited by 2
- 2026Fidelity Probes for Specification--Code Alignmentcited by 1
Built on by
Not cited yet.