# Studying the Difference Between Natural and Programming Language Corpora

arXiv: https://arxiv.org/abs/1806.02437 · HTML: https://mlsanity.com/paper/1806.02437/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jun 2018.

## Built on by (most cited first, 13 of 13)

- [Outline, Then Details: Syntactically Guided Coarse-To-Fine Code Generation](https://mlsanity.com/paper/2305.00909/) (2023)
- [Learning to Complete Code with Sketches](https://mlsanity.com/paper/2106.10158/) (2021)
- [Memorization and Generalization in Neural Code Intelligence Models](https://mlsanity.com/paper/2106.08704/) (2021)
- [An Exploratory Study on Code Attention in BERT](https://mlsanity.com/paper/2204.10200/) (2022)
- [Studying LLM Performance on Closed- and Open-source Data](https://mlsanity.com/paper/2402.15100/) (2024)
- [A Critical Study of What Code-LLMs (Do Not) Learn](https://mlsanity.com/paper/2406.11930/) (2024)
- [Scaling Laws for Code: A More Data-Hungry Regime](https://mlsanity.com/paper/2510.08702/) (2025)
- [Statistical Patterns in the Equations of Physics and the Emergence of a Meta-Law of Nature](https://mlsanity.com/paper/2408.11065/) (2024)
- [LAMNER: Code Comment Generation Using Character Language Model and Named Entity Recognition](https://mlsanity.com/paper/2204.09654/) (2022)
- [Redundancy and Concept Analysis for Code-trained Language Models](https://mlsanity.com/paper/2305.00875/) (2023)
- [AnCoder: Anchored Code Generation via Discrete Diffusion Models](https://mlsanity.com/paper/2602.17688/) (2026)
- [Position: Natural Language Should Not Fully Replace Formal Languages](https://mlsanity.com/paper/2607.20432/) (2026)
- [Uncertainty Quantification for LLM-based Code Generation](https://mlsanity.com/paper/2605.12201/) (2026)

## Abstract (arXiv)

Code corpora, as observed in large software systems, are now known to be far more repetitive and predictable than natural language corpora. But why? Does the difference simply arise from the syntactic limitations of programming languages? Or does it arise from the differences in authoring decisions made by the writers of these natural and programming language texts? We conjecture that the differences are not entirely due to syntax, but also from the fact that reading and writing code is un-natural for humans, and requires substantial mental effort; so, people prefer to write code in ways that are familiar to both reader and writer. To support this argument, we present results from two sets of studies: 1) a first set aimed at attenuating the effects of syntax, and 2) a second, aimed at measuring repetitiveness of text written in other settings (e.g. second language, technical/specialized jargon), which are also effortful to write. We find find that this repetition in source code is not entirely the result of grammar constraints, and thus some repetition must result from human choice. While the evidence we find of similar repetitive behavior in technical and learner corpora does not conclusively show that such language is used by humans to mitigate difficulty, it is consistent with that theory.