# Rethinking Communication Metrics: How Should We Measure Meaning?

arXiv: https://arxiv.org/abs/2608.21626 · HTML: https://mlsanity.com/paper/2608.21626/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 37 references that are in the map.
- First posted on arXiv 21 Aug 2026.

## Builds on (most cited first, 19 of 19)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Deep Learning Enabled Semantic Communication Systems](https://mlsanity.com/paper/2006.10685/) (2020)
- [Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communications](https://mlsanity.com/paper/2207.09353/) (2022)
- [Learning Task-Oriented Communication for Edge Inference: An Information Bottleneck Approach](https://mlsanity.com/paper/2102.04170/) (2021)
- [DeepJSCC-f: Deep Joint Source-Channel Coding of Images with Feedback](https://mlsanity.com/paper/1911.11174/) (2019)
- [Semantic Communications for Future Internet: Fundamentals, Applications, and Challenges](https://mlsanity.com/paper/2207.00427/) (2022)
- [Nonlinear Transform Source-Channel Coding for Semantic Communications](https://mlsanity.com/paper/2112.10961/) (2021)
- [What Will 5G Be?](https://mlsanity.com/paper/1405.2957/) (2014)
- [Key Performance Indicators for QOS Assessment in TETRA Networks](https://mlsanity.com/paper/1401.1918/) (2014)

## Abstract (arXiv)

Semantic communication shifts the objective of communication systems from accurate symbol reconstruction toward meaning preservation, task accomplishment, and efficient information exchange. However, its evaluation remains fragmented across telecommunications, natural language processing, computer vision, and machine learning, and no single metric can characterize semantic quality across modalities, tasks, and channel conditions. This article surveys key performance indicators (KPIs) for text- and image-based semantic communication systems from a unified, evaluation-centered perspective. Unlike prior surveys primarily organized around architectures, applications, or transmission strategies, this work focuses on how semantic success should be defined and measured. Existing KPIs are classified according to communication goal, source modality, receiver output, reference availability, evaluation level, and channel or resource constraints. The survey reviews reconstruction-based, task-oriented, reference-free, representation-level, perceptual, and channel-aware metrics, and presents a cross-modality comparison of their roles, strengths, and limitations. It further analyzes how unresolved semantic-KPI challenges affect monitoring, quality assurance, resource optimization, fault diagnosis, and standardization. Key open problems include the absence of universal semantic success criteria and standardized semantic ground truth, semantic drift, limited reference-free evaluation, weak integration of machine-learning metrics with communication constraints, and the lack of relation-level and multimodal KPIs. Finally, future research directions are outlined toward standardized, interpretable, adaptive, task-aware, and communication-aware evaluation frameworks.