mlsanityData to 7 Oct 2026

Spaghetti Architect: A Contamination-Resistant, By-Construction-Labelled, Multi-Language Code Dataset Generator

arXiv 2607.18642 ↗First posted 21 Jul 2026cs.LG cs.SEcited by 0

Not on the map. Why?

Abstract (arXiv)

Mined code corpora are abundant but uncontrolled: a snippet's semantics, surface "messiness," and difficulty are whatever the wild contained; there is no known-optimal reference to grade against; and any public sample may already sit in a model's training set. We present Spaghetti Architect, a tool that mints code datasets with the control such corpora lack. An anti-optimization transpiler maps a clean, language-agnostic JSON intermediate representation to deliberately redundant, fully-flattened programs in five languages (Python, JavaScript, Go, Java, C++); every program is compiled, run, and checked against a reference oracle, so each instance is correct by construction. The clean IR is a known-optimal reference, messiness is dialed by strictly-nested anti-pattern profiles, each instance is labelled along two orthogonal difficulty axes, intrinsic (problem size) and incidental (presentation at fixed semantics), and contamination is resisted by minting fresh variants from a private held-out seed. We give construct-validity evidence that the quality order moves established complexity and readability metrics, and report baselines on a four-model open ladder: exact match rises with scale, and the intrinsic knob collapses arithmetic-aggregation accuracy of even the strongest model to zero. Further, development-set scores equal freshly re-minted held-out counterparts within |Delta|≤ 0.012 (comprehension) and ≤ 0.011 (refactoring); on identical programs, refactoring equivalence (0.73 rightarrow 0.99) is scale-invariant while output prediction collapses; and ablating the generator's self-annotations shows they inflate the weakest model an order of magnitude more than the strongest (-0.173 vs -0.017): the annotated ladder resolves one of three adjacent pairs where the unannotated resolves all three. Open source (MIT), dependency-free, archived under a persistent DOI.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 32 of its 43 references that are in the map. First posted on arXiv 21 Jul 2026.

Builds on

32 of 43 references.

References · 32

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2021Program Synthesis with Large Language Modelscited by 3,509
  3. 2020Extracting Training Data from Large Language Modelscited by 2,144
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  5. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  6. 2018Datasheets for Datasetscited by 1,371
  7. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  8. 2018Model Cards for Model Reportingcited by 1,068
  9. 2021Measuring Coding Challenge Competence With APPScited by 879
  10. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  11. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  12. 2023Textbooks Are All You Needcited by 561
20 more
  1. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 533
  2. 2023Detecting Pretraining Data from Large Language Modelscited by 375
  3. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
  4. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 309
  5. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 295
  6. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
  7. 2020Unsupervised Translation of Programming Languagescited by 262
  8. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 251
  9. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 242
  10. 2023Proving Test Set Contamination in Black Box Language Modelscited by 227
  11. 2023Inverse Scaling: When Bigger Isn't Bettercited by 205
  12. 2021CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Taskscited by 199
  13. 2022Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AIcited by 169
  14. 2023Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarkscited by 139
  15. 2022ReCode: Robustness Evaluation of Code Generation Modelscited by 100
  16. 2023DyVal: Dynamic Evaluation of Large Language Models for Reasoning Taskscited by 100
  17. 2022Natural Attack for Pre-trained Models of Codecited by 72
  18. 2024Quantifying Contamination in Evaluating Code Generation Capabilities of Language Modelscited by 62
  19. 2023CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translationcited by 55
  20. 2024Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLMcited by 38

Built on by

Not cited yet.