# GAIT: A Geometric Approach to Information Theory

arXiv: https://arxiv.org/abs/1906.08325 · HTML: https://mlsanity.com/paper/1906.08325/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Jun 2019.

## Built on by (most cited first, 5 of 5)

- [The Vendi Score: A Diversity Evaluation Metric for Machine Learning](https://mlsanity.com/paper/2210.02410/) (2022)
- [Geometric Entropic Exploration](https://mlsanity.com/paper/2101.02055/) (2021)
- [Loss Functions for Classification using Structured Entropy](https://mlsanity.com/paper/2206.07122/) (2022)
- [Distributionally Robust Geometric Joint Chance-Constrained Optimization: Neurodynamic Approaches](https://mlsanity.com/paper/2603.06597/) (2026)
- [Representation of syntax in LLMs through the lens of linear distance and similarity-aware entropy](https://mlsanity.com/paper/2608.27813/) (2026)

## Abstract (arXiv)

We advocate the use of a notion of entropy that reflects the relative abundances of the symbols in an alphabet, as well as the similarities between them. This concept was originally introduced in theoretical ecology to study the diversity of ecosystems. Based on this notion of entropy, we introduce geometry-aware counterparts for several concepts and theorems in information theory. Notably, our proposed divergence exhibits performance on par with state-of-the-art methods based on the Wasserstein distance, but enjoys a closed-form expression that can be computed efficiently. We demonstrate the versatility of our method via experiments on a broad range of domains: training generative models, computing image barycenters, approximating empirical measures and counting modes.