Style, Not Self: Surface Cues Explain Zero-Shot Code Attribution by Large Language Models
Not on the map. Why?
Abstract (arXiv)
If a language model can recognize code it wrote, it may favor that code as a judge, and instances of one model monitoring each other could collude. We test this zero-shot on current commercial models. Five LLMs generate solutions to MBPP, HumanEval, and DS-1000, seven more to MBPP, and models act as evaluators in four tasks: picking their own solution from a pair, judging whether a single solution is their own, identifying which of two solutions a named model wrote, and judging quality blind. In the single-solution task, balanced accuracy is 49-58% for all 15 model-benchmark combinations, while raw accuracy (38-67%) mostly reflects how readily a model claims authorship. In the pairwise task, accuracy across 14 evaluator-opponent combinations correlates at r=0.93 with how often the evaluator's solution is longer. Attribution to a named model succeeds on some pairs and is consistently inverted on others. A rule-based normalization that strips docstrings, comments, type hints, and local names preserves Pass@1 and leaves ten of twelve re-tested results at chance; the other two follow a length difference it leaves, although a trained classifier still separates most normalized pairs. Claude Haiku's self-preference also disappears. We recommend reporting balanced accuracy, heuristic baselines, and label consistency.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 22 of its 30 references that are in the map. First posted on arXiv 24 Sep 2026.
Builds on
22 of 30 references.
References · 22
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2021Program Synthesis with Large Language Modelscited by 3,509
- 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
- 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
- 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
- 2024Self-Preference Bias in LLM-as-a-Judgecited by 248
- 2023AI Control: Improving Safety Despite Intentional Subversioncited by 207
- 2024Secret Collusion among AI Agents: Multi-Agent Deception via Steganographycited by 107
- 2024Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMscited by 78
- 2024Hidden in Plain Text: Emergence & Mitigation of Steganographic Collusion in LLMscited by 52
- 2019Misleading Authorship Attribution of Source Code using Adversarial Learningcited by 33
10 more
- 2024Your Large Language Models Are Leaving Fingerprintscited by 31
- 2023Zero-Shot Detection of Machine-Generated Codescited by 24
- 2023A Ship of Theseus: Curious Cases of Paraphrasing in LLM-Generated Textscited by 23
- 2024StyleRemix: Interpretable Authorship Obfuscation via Distillation and Perturbation of Style Elementscited by 18
- 2025Don't Judge Code by Its Cover: Exploring Biases in LLM Judges for Code Evaluationcited by 18
- 2024Inspection and Control of Self-Generated-Text Recognition Ability in Llama3-8b-Instructcited by 15
- 2026When can we trust untrusted monitoring? A safety case sketch across collusion strategiescited by 9
- 2025Know Thyself? On the Incapability and Implications of AI Self-Recognitioncited by 7
- 2025Can AI Recognize Its Own Reflection? Self-Detection Performance of LLMs in Computing Educationcited by 3
- 2026AICD Bench: A Challenging Benchmark for AI-Generated Code Detectioncited by 2
Built on by
Not cited yet.