# Markov Categories and Entropy

arXiv: https://arxiv.org/abs/2212.11719 · HTML: https://mlsanity.com/paper/2212.11719/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Dec 2022.

## Built on by (most cited first, 7 of 7)

- [Neural Circuit Diagrams: Robust Diagrams for the Communication, Implementation, and Analysis of Deep Learning Architectures](https://mlsanity.com/paper/2402.05424/) (2024)
- [FlashAttention on a Napkin: A Diagrammatic Approach to Deep Learning IO-Awareness](https://mlsanity.com/paper/2412.03317/) (2024)
- ["X of Information'' Continuum: A Survey on AI-Driven Multi-dimensional Metrics for Next-Generation Networked Systems](https://mlsanity.com/paper/2507.19657/) (2025)
- [Active Inference in String Diagrams: A Categorical Account of Predictive Processing and Free Energy](https://mlsanity.com/paper/2308.00861/) (2023)
- [Enriching Disentanglement: From Logical Definitions to Quantitative Metrics](https://mlsanity.com/paper/2305.11512/) (2023)
- [A Markov Categorical Framework for Language Modeling](https://mlsanity.com/paper/2507.19247/) (2025)
- [Transparent Semantic Spaces: A Categorical Approach to Explainable Word Embeddings](https://mlsanity.com/paper/2508.20701/) (2025)

## Abstract (arXiv)

Markov categories are a novel framework to describe and treat problems in probability and information theory. In this work we combine the categorical formalism with the traditional quantitative notions of entropy, mutual information, and data processing inequalities. We show that several quantitative aspects of information theory can be captured by an enriched version of Markov categories, where the spaces of morphisms are equipped with a divergence or even a metric. As it is customary in information theory, mutual information can be defined as a measure of how far a joint source is from displaying independence of its components. More strikingly, Markov categories give a notion of determinism for sources and channels, and we can define entropy exactly by measuring how far a source or channel is from being deterministic. This recovers Shannon and R'enyi entropies, as well as the Gini-Simpson index used in ecology to quantify diversity, and it can be used to give a conceptual definition of generalized entropy.