Co-Located Tests, Better AI Code: How Test Syntax Structure Affects Foundation Model Code Generation
Not on the map. Why?
Abstract (arXiv)
AI coding assistants increasingly generate code alongside tests. How developers structure test code, whether inline with the implementation or in separate blocks, has traditionally been a matter of testing philosophy. We investigate whether this choice affects AI code generation quality. We conduct a large-scale empirical study (830+ generated files, 12 models, 3 providers) using SEGA, a three-dimensional evaluation framework measuring Determinism, Preservation, and Correctness. Comparing inline test syntax (Python doctests) against separated test syntax (Rust #[test] blocks) on a d-ary heap implementation, we find that: (1) inline tests yield near-perfect preservation (100%) and correctness (92-100%) across all models; (2) separated tests expose stark model-tier gaps (0-100% correctness) and independence between preservation and correctness; (3) model behavior evolves across generations, and notably one model breaks the test suppression pattern of its three predecessors; (4) mechanistic analysis on 7 open-source architectures (6 transformers and a gated-linear Recurrent Neural Network (RNN)) reveals inline test markers receive 2.8-4.4× stronger attention in 5/7 models, with causal validation via knockout and steering experiments on the 4 code-specialized transformers and RWKV-6; the co-location mechanism extends to a non-transformer architecture, suggesting the design recommendation is robust to future architectural shifts. In the Foundation Model era, test syntax structure is a software design concern: co-locating tests with implementation code produces measurably better AI-generated code. This arxiv long version includes appendices that further qualify the effect as bounded by both model capability and programming language.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 23 of its 50 references that are in the map. First posted on arXiv 20 Apr 2026.
Builds on
23 of 50 references.
References · 23
- 2017Attention Is All You Needcited by 60,361
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2022Locating and Editing Factual Associations in GPTcited by 2,691
- 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
- 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
- 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
- 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
- 2024StarCoder 2 and The Stack v2: The Next Generationcited by 502
- 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 431
- 2022CodeT: Code Generation with Generated Testscited by 384
- 2023The Impact of AI on Developer Productivity: Evidence from GitHub Copilotcited by 274
- 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 243
11 more
- 2023An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generationcited by 194
- 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 159
- 2022DocPrompting: Generating Code by Retrieving the Docscited by 153
- 2024CodeGemma: Open Code Models Based on Gemmacited by 150
- 2023An Empirical Study of the Non-determinism of ChatGPT in Code Generationcited by 137
- 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 136
- 2023No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generationcited by 82
- 2024Bugs in Large Language Models Generated Code: An Empirical Studycited by 60
- 2024What's Wrong with Your Code Generated by Large Language Models? An Extensive Studycited by 53
- 2022What Do They Capture? -- A Structural Analysis of Pre-Trained Language Models for Source Codecited by 50
- 2021What do pre-trained code models know about code?cited by 36
Built on by
Not cited yet.