# Formal Aspects of Language Modeling

arXiv: https://arxiv.org/abs/2311.04329 · HTML: https://mlsanity.com/paper/2311.04329/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Nov 2023.

## Built on by (most cited first, 7 of 7)

- [Gumbel Counterfactual Generation From Language Models](https://mlsanity.com/paper/2411.07180/) (2024)
- [Autoregressive Language Models are Secretly Energy-Based Models: Insights into the Lookahead Capabilities of Next-Token Prediction](https://mlsanity.com/paper/2512.15605/) (2025)
- [On the Reasoning Abilities of Masked Diffusion Language Models](https://mlsanity.com/paper/2510.13117/) (2025)
- [Language Models Are Implicitly Continuous](https://mlsanity.com/paper/2504.03933/) (2025)
- [Unraveling Syntax: Language Modeling and the Substructure of Grammars](https://mlsanity.com/paper/2510.02524/) (2025)
- [Revisiting Padded Transformer Expressivity: Which Architectural Choices Matter and Which Don't](https://mlsanity.com/paper/2605.30523/) (2026)
- [A fine-grained look at causal effects in causal spaces](https://mlsanity.com/paper/2512.11919/) (2025)

## Abstract (arXiv)

Large language models have become one of the most commonly deployed NLP inventions. In the past half-decade, their integration into core natural language processing tools has dramatically increased the performance of such tools, and they have entered the public discourse surrounding artificial intelligence. Consequently, it is important for both developers and researchers alike to understand the mathematical foundations of large language models, as well as how to implement them. These notes are the accompaniment to the theoretical portion of the ETH Z"urich course on large language models, covering what constitutes a language model from a formal, theoretical perspective.