# Attention to Entropic Communication

arXiv: https://arxiv.org/abs/2307.11423 · HTML: https://mlsanity.com/paper/2307.11423/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 73 references that are in the map.
- First posted on arXiv 21 Jul 2023.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Introduction to Variational Autoencoders](https://mlsanity.com/paper/1906.02691/) (2019)
- [ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering](https://mlsanity.com/paper/1511.05960/) (2015)
- [Metric Gaussian Variational Inference](https://mlsanity.com/paper/1901.11033/) (2019)
- [Geometric variational inference](https://mlsanity.com/paper/2105.10470/) (2021)
- [Optimal Belief Approximation](https://mlsanity.com/paper/1610.09018/) (2016)
- [Probabilistic Autoencoder using Fisher Information](https://mlsanity.com/paper/2110.14947/) (2021)
- [Normalized Power Prior Bayesian Analysis](https://mlsanity.com/paper/2204.05615/) (2022)
- [A Reputation Game Simulation: Emergent Social Phenomena from Information Theory](https://mlsanity.com/paper/2106.05414/) (2021)

## Abstract (arXiv)

The concept of attention, numerical weights that emphasize the importance of particular data, has proven to be very relevant in artificial intelligence. Relative entropy (RE, aka Kullback-Leibler divergence) plays a central role in communication theory. Here we combine these concepts, attention and RE. RE guides optimal encoding of messages in bandwidth-limited communication as well as optimal message decoding via the maximum entropy principle (MEP). In the coding scenario, RE can be derived from four requirements, namely being analytical, local, proper, and calibrated. Weighted RE, used for attention steering in communications, turns out to be improper. To see how proper attention communication can emerge, we analyze a scenario of a message sender who wants to ensure that the receiver of the message can perform well-informed actions. If the receiver decodes the message using the MEP, the sender only needs to know the receiver's utility function to inform optimally, but not the receiver's initial knowledge state. In case only the curvature of the utility function maxima are known, it becomes desirable to accurately communicate an attention function, in this case a by this curvature weighted and re-normalized probability function. Entropic attention communication is here proposed as the desired generalization of entropic communication that permits weighting while being proper, thereby aiding the design of optimal communication protocols in technical applications and helping to understand human communication. For example, our analysis shows how to derive the level of cooperation expected under misaligned interests of otherwise honest communication partners.