# Beyond Fidelity: Semantic Similarity Assessment in Low-Level Image Processing

arXiv: https://arxiv.org/abs/2604.25408 · HTML: https://mlsanity.com/paper/2604.25408/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 39 references that are in the map.
- First posted on arXiv 28 Apr 2026.

## Builds on (most cited first, 18 of 18)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Image Quality Assessment: Unifying Structure and Texture Similarity](https://mlsanity.com/paper/2004.07728/) (2020)
- [Deep Neural Networks for No-Reference and Full-Reference Image Quality Assessment](https://mlsanity.com/paper/1612.01697/) (2016)
- [Diffusion Models for Image Restoration and Enhancement: A Comprehensive Survey](https://mlsanity.com/paper/2308.09388/) (2023)
- [Perceive-IR: Learning to Perceive Degradation Better for All-in-One Image Restoration](https://mlsanity.com/paper/2408.15994/) (2024)
- [Diff-Plugin: Revitalizing Details for Diffusion-based Low-level Tasks](https://mlsanity.com/paper/2403.00644/) (2024)
- [Towards Semantic Communications: Deep Learning-Based Image Semantic Coding](https://mlsanity.com/paper/2208.04094/) (2022)
- [MB-TaylorFormer V2: Improved Multi-branch Linear Transformer Expanded by Taylor Formula for Image Restoration](https://mlsanity.com/paper/2501.04486/) (2025)
- [SGUIE-Net: Semantic Attention Guided Underwater Image Enhancement with Multi-Scale Perception](https://mlsanity.com/paper/2201.02832/) (2022)
- [Multimodal LLMs Can Reason about Aesthetics in Zero-Shot](https://mlsanity.com/paper/2501.09012/) (2025)
- [Distilling Knowledge from Object Classification to Aesthetics Assessment](https://mlsanity.com/paper/2206.00809/) (2022)
- [Image Quality Assessment: From Human to Machine Preference](https://mlsanity.com/paper/2503.10078/) (2025)
- [Semantic Similarity Score for Measuring Visual Similarity at Semantic Level](https://mlsanity.com/paper/2406.03865/) (2024)
- [All-in-One Image Restoration via Causal-Deconfounding Wavelet-Disentangled Prompt Network](https://mlsanity.com/paper/2603.03839/) (2026)

## Abstract (arXiv)

Low-level image processing has long been evaluated mainly from the perspective of visual fidelity. However, with the rise of deep learning and generative models, processed images may preserve perceptual quality while altering semantic content, making conventional Image Quality Assessment (IQA) insufficient for semantic-level assessment. In this paper, we formalize textitSemantic Similarity as a new evaluation task for low-level image processing, aimed at measuring whether semantic content is preserved after processing. We further present a structured formulation of image semantics based on semantic entities and their relations, and discuss the desired properties and constraints of a valid semantic similarity index. Based on this formulation, we propose Triplet-based Semantic Similarity Score (T3S), which models image semantics through foreground entities, background entities, and relations. T3S combines semantic entity extraction, foreground-background disentanglement, and open-world class/relation modeling. Experiments on COCO and SPA-Data show that T3S consistently outperforms existing fidelity-oriented metrics and representative semantic-level baselines, while better reflecting progressive semantic changes under diverse degradations. These results highlight the importance of semantic assessment in modern low-level vision.