# How to Evaluate Semantic Communications for Images with ViTScore Metric? arXiv: https://arxiv.org/abs/2309.04891 ยท HTML: https://mlsanity.com/paper/2309.04891/ ## Key facts - Cited by 6 papers in the map (data through 14 Oct 2026). - Builds on 41 of its 69 references that are in the map. - First posted on arXiv 9 Sep 2023. ## Builds on (most cited first, 30 of 41) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020) - [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019) - [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023) - [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014) - [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018) - [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019) - [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013) - [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021) - [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014) - [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016) - [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015) - [The Perception-Distortion Tradeoff](https://mlsanity.com/paper/1711.06077/) (2017) - [In-Domain GAN Inversion for Real Image Editing](https://mlsanity.com/paper/2004.00049/) (2020) - [GAN Inversion: A Survey](https://mlsanity.com/paper/2101.05278/) (2021) - [Deep Learning Enabled Semantic Communication Systems](https://mlsanity.com/paper/2006.10685/) (2020) - [An Introduction to Deep Learning for the Physical Layer](https://mlsanity.com/paper/1702.00832/) (2017) - [Learning Fine-grained Image Similarity with Deep Ranking](https://mlsanity.com/paper/1404.4661/) (2014) - [Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communications](https://mlsanity.com/paper/2207.09353/) (2022) - [Visual Translation Embedding Network for Visual Relation Detection](https://mlsanity.com/paper/1702.08319/) (2017) - [Semantic Communications: Principles and Challenges](https://mlsanity.com/paper/2201.01389/) (2021) - [Nonlinear Transform Source-Channel Coding for Semantic Communications](https://mlsanity.com/paper/2112.10961/) (2021) - [Task-Oriented Multi-User Semantic Communications](https://mlsanity.com/paper/2112.10255/) (2021) - [Wireless Image Retrieval at the Edge](https://mlsanity.com/paper/2007.10915/) (2020) - [Robust Semantic Communications with Masked VQ-VAE Enabled Codebook](https://mlsanity.com/paper/2206.04011/) (2022) - [Toward a 6G AI-Native Air Interface](https://mlsanity.com/paper/2012.08285/) (2020) - [From Semantic Communication to Semantic-aware Networking: Model, Architecture, and Open Problems](https://mlsanity.com/paper/2012.15405/) (2020) - [Task-Oriented Multi-User Semantic Communications for VQA Task](https://mlsanity.com/paper/2108.07357/) (2021) - [Bandwidth-Agile Image Transmission with Deep Joint Source-Channel Coding](https://mlsanity.com/paper/2009.12480/) (2020) - [Towards Semantic Communications: Deep Learning-Based Image Semantic Coding](https://mlsanity.com/paper/2208.04094/) (2022) ## Built on by (most cited first, 6 of 6) - [Semantic Similarity Score for Measuring Visual Similarity at Semantic Level](https://mlsanity.com/paper/2406.03865/) (2024) - [Semantic Edge Computing and Semantic Communications in 6G Networks: A Unifying Survey and Research Challenges](https://mlsanity.com/paper/2411.18199/) (2024) - [Semantic Edge-Cloud Communication for Real-Time Urban Traffic Surveillance with ViT and LLMs over Mobile Networks](https://mlsanity.com/paper/2509.21259/) (2025) - [Human-Aligned Procedural Level Generation Reinforcement Learning via Text-Level-Sketch Shared Representation](https://mlsanity.com/paper/2508.09860/) (2025) - [MRD: Using Physically Based Differentiable Rendering to Probe Vision Models for 3D Scene Understanding](https://mlsanity.com/paper/2512.12307/) (2025) - [Multiverse: Language-Conditioned Multi-Game Level Blending via Shared Representation](https://mlsanity.com/paper/2603.26782/) (2026) ## Abstract (arXiv) Semantic communications (SC) have been expected to be a new paradigm shifting to catalyze the next generation communication, whose main concerns shift from accurate bit transmission to effective semantic information exchange in communications. However, the previous and widely-used metrics for images are not applicable to evaluate the image semantic similarity in SC. Classical metrics to measure the similarity between two images usually rely on the pixel level or the structural level, such as the PSNR and the MS-SSIM. Straightforwardly using some tailored metrics based on deep-learning methods in CV community, such as the LPIPS, is infeasible for SC. To tackle this, inspired by BERTScore in NLP community, we propose a novel metric for evaluating image semantic similarity, named Vision Transformer Score (ViTScore). We prove theoretically that ViTScore has 3 important properties, including symmetry, boundedness, and normalization, which make ViTScore convenient and intuitive for image measurement. To evaluate the performance of ViTScore, we compare ViTScore with 3 typical metrics (PSNR, MS-SSIM, and LPIPS) through 4 classes of experiments: (i) correlation with BERTScore through evaluation of image caption downstream CV task, (ii) evaluation in classical image communications, (iii) evaluation in image semantic communication systems, and (iv) evaluation in image semantic communication systems with semantic attack. Experimental results demonstrate that ViTScore is robust and efficient in evaluating the semantic similarity of images. Particularly, ViTScore outperforms the other 3 typical metrics in evaluating the image semantic changes by semantic attack, such as image inverse with Generative Adversarial Networks (GANs). This indicates that ViTScore is an effective performance metric when deployed in SC scenarios.