# How to select an objective function using information theory

arXiv: https://arxiv.org/abs/2212.06566 · HTML: https://mlsanity.com/paper/2212.06566/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 1 of its 53 references that are in the map.
- First posted on arXiv 10 Dec 2022.

## Builds on (most cited first, 1 of 1)

- [On the interplay between noise and curvature and its effect on optimization and generalization](https://mlsanity.com/paper/1906.07774/) (2019)

## Abstract (arXiv)

In machine learning or scientific computing, model performance is measured with an objective function. But why choose one objective over another? Information theory gives one answer: To maximize the information in the model, select the objective function that represents the error in the fewest bits. To evaluate different objectives, transform them into likelihood functions. As likelihoods, their relative magnitude represents how strongly we should prefer one objective versus another, and the log of that relation represents the difference in their bit-length, as well as the difference in their uncertainty. In other words, prefer whichever objective minimizes the uncertainty. Under the information-theoretic paradigm, the ultimate objective is to maximize information (and minimize uncertainty), as opposed to any specific utility. We argue that this paradigm is well-suited to models that have many uses and no definite utility, like the large Earth system models used to understand the effects of climate change.