mlsanityData to 7 Oct 2026

Measuring What a Specification Determines: A Formal Semantic-Block Model and an Execution-Judged Benchmark

arXiv 2608.19475 ↗First posted 19 Aug 2026cs.SE cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

This work introduces a formal semantic-block model for specifications and an execution-judged benchmark for evaluating specification quality independently of model capability. A specification is represented as a structure comprising semantic blocks, dependency relations, block-owned rules, decision points, and explicitly open questions, subject to four machine-checkable well-formedness conditions: acyclicity, single ownership, constraint domination, and totality or ambiguity-stop. Determinacy is defined model-theoretically as agreement among all conforming implementations and is estimated empirically through convergence across independent implementers. The model is instantiated on an Oracle-to-PostgreSQL migration specification containing 18 blocks and 19 dependency edges. Computational validation shows that the five-layer decomposition reduces mean per-task context by approximately 71% through dependency closures, covers 85.5% of the study-defined Oracle construct taxonomy with all identified gaps triaged, is not Pareto-dominated by the tested alternative partitions, and is recovered at the 99.9th percentile from citation-derived edges not used to define the original structure. The benchmark keeps the implementer panel fixed, includes a mandatory no-specification control arm, and uses PostgreSQL 16 and a live Oracle instance as deterministic execution judges. Six designed studies, including three pre-registered manipulations and three diagnostic analyses, further examine specification effects. Repeated runs on a 25-unit subsample reveal an empirical variability floor with a median arm-delta spread of 14.4 percentage points. The results support determinacy as a formal concept but not as a standalone empirical quality metric for the evaluated contemporary LLM implementers.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 20 of its 37 references that are in the map. First posted on arXiv 19 Aug 2026.

Builds on

20 of 37 references.

References · 20

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  2. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  3. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
  4. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 417
  5. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  6. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  7. 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 223
  8. 2023An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generationcited by 193
  9. 2022Do Users Write More Insecure Code with AI Assistants?cited by 186
  10. 2023Large Language Models for Software Engineering: Survey and Open Problemscited by 178
  11. 2020Semantic Evaluation for Text-to-SQL with Distilled Test Suitescited by 123
  12. 2024On the Evaluation of Large Language Models in Unit Test Generationcited by 33
8 more
  1. 2026Spec-Driven Development:From Code to Contract in the Age of AI Coding Assistantscited by 14
  2. 2016Rapid quality assurance with Requirements Smellscited by 11
  3. 2026Specification-Based Code-Text-Code Reengineering for LLM-Mediated Software Evolutioncited by 5
  4. 2026Unified Architecture Metamodel of Information Systems Developed by Generative AIcited by 4
  5. 2026Token Optimization Strategies for LLM-Based Oracle-to-PostgreSQL Migrationcited by 3
  6. 2025Fine-tuned LLM-based Code Migration Frameworkcited by 2
  7. 2026Specification-Driven Development Benchmark: Security Knowledge Transitioncited by 2
  8. 2026Fidelity Probes for Specification--Code Alignmentcited by 1

Built on by

Not cited yet.