# On the "Naturalness" of Buggy Code

arXiv: https://arxiv.org/abs/1506.01159 · HTML: https://mlsanity.com/paper/1506.01159/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jun 2015.

## Built on by (most cited first, 30 of 35)

- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [Unified Pre-training for Program Understanding and Generation](https://mlsanity.com/paper/2103.06333/) (2021)
- [Keep the Conversation Going: Fixing 162 out of 337 bugs for 0.42 each using ChatGPT](https://mlsanity.com/paper/2304.00385/) (2023)
- [Large Language Models for Test-Free Fault Localization](https://mlsanity.com/paper/2310.01726/) (2023)
- [Self-Supervised Bug Detection and Repair](https://mlsanity.com/paper/2105.12787/) (2021)
- [NatGen: Generative pre-training by "Naturalizing" source code](https://mlsanity.com/paper/2206.07585/) (2022)
- [On Multi-Modal Learning of Editing Source Code](https://mlsanity.com/paper/2108.06645/) (2021)
- [Understanding Neural Code Intelligence Through Program Simplification](https://mlsanity.com/paper/2106.03353/) (2021)
- [EpiCoder: Encompassing Diversity and Complexity in Code Generation](https://mlsanity.com/paper/2501.04694/) (2025)
- [Can LLMs Reason About Program Semantics? A Comprehensive Evaluation of LLMs on Formal Specification Inference](https://mlsanity.com/paper/2503.04779/) (2025)
- [Toward a Theory of Causation for Interpreting Neural Code Models](https://mlsanity.com/paper/2302.03788/) (2023)
- [Detect-Localize-Repair: A Unified Framework for Learning to Debug with CodeT5](https://mlsanity.com/paper/2211.14875/) (2022)
- [Exploring Code Language Models for Automated HLS-based Hardware Generation: Benchmark, Infrastructure and Analysis](https://mlsanity.com/paper/2502.13921/) (2025)
- [Towards a consistent interpretation of AIOps models](https://mlsanity.com/paper/2202.02298/) (2022)
- [Show Me Your Code! Kill Code Poisoning: A Lightweight Method Based on Code Naturalness](https://mlsanity.com/paper/2502.15830/) (2025)
- [Variable Name Recovery in Decompiled Binary Code using Constrained Masked Language Modeling](https://mlsanity.com/paper/2103.12801/) (2021)
- [Towards Reliable Evaluation of Neural Program Repair with Natural Robustness Testing](https://mlsanity.com/paper/2402.11892/) (2024)
- [Leveraging Reviewer Experience in Code Review Comment Generation](https://mlsanity.com/paper/2409.10959/) (2024)
- [ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Grounding](https://mlsanity.com/paper/2504.00019/) (2025)
- [Learning to Extend Program Graphs to Work-in-Progress Code](https://mlsanity.com/paper/2105.14038/) (2021)
- [Assessing the Latent Automated Program Repair Capabilities of Large Language Models using Round-Trip Translation](https://mlsanity.com/paper/2401.07994/) (2024)
- [Aligning Crowd-sourced Human Feedback for Reinforcement Learning on Code Generation by Large Language Models](https://mlsanity.com/paper/2503.15129/) (2025)
- [The Fools are Certain; the Wise are Doubtful: Exploring LLM Confidence in Code Completion](https://mlsanity.com/paper/2508.16131/) (2025)
- [Leveraging Structural Properties of Source Code Graphs for Just-In-Time Bug Prediction](https://mlsanity.com/paper/2201.10137/) (2022)
- [INSPECT: Intrinsic and Systematic Probing Evaluation for Code Transformers](https://mlsanity.com/paper/2312.05092/) (2023)
- [Deep Learning-Based Identification of Inconsistent Method Names: How Far Are We?](https://mlsanity.com/paper/2501.12617/) (2025)
- [Deep Learning-based Intrusion Detection Systems: A Survey](https://mlsanity.com/paper/2504.07839/) (2025)
- [Code for Machines, Not Just Humans: Quantifying AI-Friendliness with Code Health Metrics](https://mlsanity.com/paper/2601.02200/) (2026)
- [ANVIL: Anomaly-based Vulnerability Identification without Labelled Training Data](https://mlsanity.com/paper/2408.16028/) (2024)
- [Bringing Structure to Naturalness: On the Naturalness of ASTs](https://mlsanity.com/paper/2504.08234/) (2025)

## Abstract (arXiv)

Real software, the kind working programmers produce by the kLOC to solve real-world problems, tends to be "natural", like speech or natural language; it tends to be highly repetitive and predictable. Researchers have captured this naturalness of software through statistical models and used them to good effect in suggestion engines, porting tools, coding standards checkers, and idiom miners. This suggests that code that appears improbable, or surprising, to a good statistical language model is "unnatural" in some sense, and thus possibly suspicious. In this paper, we investigate this hypothesis. We consider a large corpus of bug fix commits (ca.~8,296), from 10 different Java projects, and we focus on its language statistics, evaluating the naturalness of buggy code and the corresponding fixes. We find that code with bugs tends to be more entropic (i.e., unnatural), becoming less so as bugs are fixed. Focusing on highly entropic lines is similar in cost-effectiveness to some well-known static bug finders (PMD, FindBugs) and ordering warnings from these bug finders using an entropy measure improves the cost-effectiveness of inspecting code implicated in warnings. This suggests that entropy may be a valid language-independent and simple way to complement the effectiveness of PMD or FindBugs, and that search-based bug-fixing methods may benefit from using entropy both for fault-localization and searching for fixes.