mlsanityData to 14 Oct 2026

The Complexity Kink: A Prompt-Side Structural Complexity Index for Code-Generation Reliability

arXiv 2609.19616 ↗First posted 17 Sep 2026cs.LG cs.SEcited by 0

Not on the map. Why?

Abstract (arXiv)

Complexity measured from generated code is failure-dependent: a difficult prompt can yield a short failing program and be assigned low output complexity. We introduce a six-dimension prompt-side structural-complexity index scored before generation and kept separate from correctness. We select 5,000 Python prompts across six bands of a preliminary single-rater rubric. Four out-of-panel LLM raters rescore the locked prompts, giving 19,997 score rows; composite inter-rater reliability is ICC = 0.872 on the 4,998 prompts with all four ratings. We evaluate 21 models per prompt, yielding 105,000 generations. In the unadjusted mean-pooled analysis, pass rate has a nonmonotone breakpoint at composite 13.75, with 79.9% at or below and 87.6% above. This is not a universal failure cutoff. Task-type fixed effects shift the breakpoint to 10.75 and cut the regime gap from 7.6 to 2.1 points. A construction-frame control shifts it to 8.50 with a raw gap of -3.5 points, and neither frame alone reproduces the pooled +7.6-point change. Model-specific fits include 16 upward and five downward changes. A 365-prompt audit-clean extension matches the original five-model estimates at bins 15 and 16 but adds only 14 prompts above bin 16. Among zero-pass generations with computable Lizard complexity, 28.5% pair a prompt composite above 8 with output complexity at most 10. Human agreement is moderate and rater-dependent on a disagreement-enriched calibration set; paraphrase and cross-language rescoring preserve score ordering. Overidentification tests reject the joint restrictions on the six dimensions, so we treat the composite as an index and make no causal interpretation of the 2SLS estimates. The contribution is a pre-generation measurement framework and a bounded observational analysis of reliability regimes.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 7 of its 19 references that are in the map. First posted on arXiv 17 Sep 2026.

Builds on

7 of 19 references.

References · 7

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2021Program Synthesis with Large Language Modelscited by 3,637
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  6. 2019Unifying Human and Statistical Evaluation for Natural Language Generationcited by 127
  7. 2022Applications of statistical causal inference in software engineeringcited by 12

Built on by

Not cited yet.