# A Semantic Generalization of Shannon's Information Theory and Applications

arXiv: https://arxiv.org/abs/2510.15871 · HTML: https://mlsanity.com/paper/2510.15871/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 106 references that are in the map.
- First posted on arXiv 6 May 2025.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [On Mutual Information Maximization for Representation Learning](https://mlsanity.com/paper/1907.13625/) (2019)
- [Deep Learning Enabled Semantic Communication Systems](https://mlsanity.com/paper/2006.10685/) (2020)
- [Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communications](https://mlsanity.com/paper/2207.09353/) (2022)
- [6G Networks: Beyond Shannon Towards Semantic and Goal-Oriented Communications](https://mlsanity.com/paper/2011.14844/) (2020)
- [Deep Learning for Joint Source-Channel Coding of Text](https://mlsanity.com/paper/1802.06832/) (2018)
- [Less Data, More Knowledge: Building Next Generation Semantic Communication Networks](https://mlsanity.com/paper/2211.14343/) (2022)
- [Joint Source-Channel Coding: Fundamentals and Recent Progress in Practical Designs](https://mlsanity.com/paper/2409.17557/) (2024)
- [The Two Kinds of Free Energy and the Bayesian Revolution](https://mlsanity.com/paper/2004.11763/) (2020)
- [Semantic Information G Theory and Logical Bayesian Inference for Machine Learning](https://mlsanity.com/paper/1809.01577/) (2018)
- [Channels' Confirmation and Predictions' Confirmation: from the Medical Test to the Raven Paradox](https://mlsanity.com/paper/2001.07566/) (2020)
- [Causal Confirmation Measures: From Simpson's Paradox to COVID-19](https://mlsanity.com/paper/2302.09067/) (2023)
- [Reviewing Evolution of Learning Functions and Semantic Information Measures for Understanding Deep Learning](https://mlsanity.com/paper/2305.14397/) (2023)
- [Semantic Variational Bayes Based on Semantic Information G Theory for Solving Latent Variables](https://mlsanity.com/paper/2408.13122/) (2024)

## Abstract (arXiv)

Does semantic communication require a semantic information theory parallel to Shannon's information theory, or can Shannon's work be generalized for semantic communication? This paper advocates for the latter and introduces a semantic generalization of Shannon's information theory (G theory for short). The core idea is to replace the distortion constraint with the semantic constraint, achieved by utilizing a set of truth functions as a semantic channel. These truth functions enable the expressions of semantic distortion, semantic information measures, and semantic information loss. Notably, the maximum semantic information criterion is equivalent to the maximum likelihood criterion and similar to the Regularized Least Squares criterion. This paper shows G theory's applications to daily and electronic semantic communication, machine learning, constraint control, Bayesian confirmation, portfolio theory, and information value. The improvements in machine learning methods involve multilabel learning and classification, maximum mutual information classification, mixture models, and solving latent variables. Furthermore, insights from statistical physics are discussed: Shannon information is similar to free energy; semantic information to free energy in local equilibrium systems; and information efficiency to the efficiency of free energy in performing work. The paper also proposes refining Friston's minimum free energy principle into the maximum information efficiency principle. Lastly, it compares G theory with other semantic information theories and discusses its limitation in representing the semantics of complex data.