mlsanityData to 14 Oct 2026

Citation Discipline in Spec-Driven Development: A Cross-Model Empirical Study of Output Determinism and Automated Hallucination Detection in LLM-Generated Code

arXiv 2606.30689 ↗First posted 28 Jun 2026cs.SE cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Spec-Driven Development (SDD) frameworks guide Large Language Model (LLM)-powered code generation through formal specifications, yet they differ fundamentally in how they enforce traceability between requirements and generated code. This paper presents two controlled empirical studies comparing three SDD frameworks: traceSDD, which enforces mandatory per-line requirement citations using hierarchical REQ-XXX.Y.Z identifiers; Spec Kit, which uses artifact-level traceability through user stories and acceptance criteria; and OpenSpec, which relies on post-hoc external trace maps. We measure two primary outcomes across two frontier LLMs -- Claude Sonnet 4.6 (N=20, 4 conditions, 240 implementations) and GLM-5-turbo (N=50, 4 conditions, 600 implementations): output determinism (lexical similarity across independent LLM sessions) and automated hallucination detection rate (TDR). Our pre-registered analysis reveals a consistent, cross-model replicated trade-off: the uncited condition produces significantly higher determinism than the cited condition (Claude: d=-0.76, p=0.003; GLM: d=-0.72, p<0.001), while only the cited condition enables automated hallucination detection (TDR: Claude 86.4%, GLM 88.0%, vs 0% for all alternatives, FPR=0% across both studies). traceSDD (cited) significantly outperforms Spec Kit on determinism (Claude: d=0.47, p=0.049; GLM: d=0.42, p=0.003) but not OpenSpec (Claude: d=0.18, p=0.44; GLM: d=0.14, p=0.32). These findings establish that citation annotations trade determinism for verifiability, and that this trade-off generalizes across model architectures.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 13 of its 25 references that are in the map. First posted on arXiv 28 Jun 2026.

Builds on

13 of 25 references.

References · 13

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  9. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  10. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  11. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  12. 2023StarCoder: may the source be with you!cited by 799
1 more
  1. 2022Large Language Models are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Modelscited by 103

Built on by

Not cited yet.