# Image and Video Quality Assessment using Prompt-Guided Latent Diffusion Models for Cross-Dataset Generalization

arXiv: https://arxiv.org/abs/2406.04654 · HTML: https://mlsanity.com/paper/2406.04654/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 64 of its 100 references that are in the map.
- First posted on arXiv 7 Jun 2024.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 64)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [Multi-Concept Customization of Text-to-Image Diffusion](https://mlsanity.com/paper/2212.04488/) (2022)
- [MUSIQ: Multi-scale Image Quality Transformer](https://mlsanity.com/paper/2108.05997/) (2021)
- [Exploring CLIP for Assessing the Look and Feel of Images](https://mlsanity.com/paper/2207.12396/) (2022)
- [On Distillation of Guided Diffusion Models](https://mlsanity.com/paper/2210.03142/) (2022)
- [Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levels](https://mlsanity.com/paper/2312.17090/) (2023)
- [KonIQ-10k: An ecologically valid database for deep learning of blind image quality assessment](https://mlsanity.com/paper/1910.06180/) (2019)
- [Exploring Video Quality Assessment on User Generated Contents from Aesthetic and Technical Perspectives](https://mlsanity.com/paper/2211.04894/) (2022)
- [Visual-Language Prompt Tuning with Knowledge-guided Context Optimization](https://mlsanity.com/paper/2303.13283/) (2023)
- [Your Diffusion Model is Secretly a Zero-Shot Classifier](https://mlsanity.com/paper/2303.16203/) (2023)
- [Blind Image Quality Assessment Using A Deep Bilinear Convolutional Neural Network](https://mlsanity.com/paper/1907.02665/) (2019)
- [Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspective](https://mlsanity.com/paper/2303.14968/) (2023)
- [From Patches to Pictures (PaQ-2-PiQ): Mapping the Perceptual Space of Picture Quality](https://mlsanity.com/paper/1912.10088/) (2019)
- [Massive Online Crowdsourced Study of Subjective and Objective Picture Quality](https://mlsanity.com/paper/1511.02919/) (2015)
- [Generalization in diffusion models arises from geometry-adaptive harmonic representations](https://mlsanity.com/paper/2310.02557/) (2023)
- [FAST-VQA: Efficient End-to-end Video Quality Assessment with Fragment Sampling](https://mlsanity.com/paper/2207.02595/) (2022)
- [PIPAL: a Large-Scale Image Quality Assessment Dataset for Perceptual Image Restoration](https://mlsanity.com/paper/2007.12142/) (2020)
- [No-Reference Image Quality Assessment via Transformers, Relative Ranking, and Self-Consistency](https://mlsanity.com/paper/2108.06858/) (2021)
- [Quality Assessment of In-the-Wild Videos](https://mlsanity.com/paper/1908.00375/) (2019)
- [Patch-VQ: 'Patching Up' the Video Quality Problem](https://mlsanity.com/paper/2011.13544/) (2020)

## Built on by (most cited first, 3 of 3)

- [LMM-VQA: Advancing Video Quality Assessment with Large Multimodal Models](https://mlsanity.com/paper/2408.14008/) (2024)
- [Revisiting Vision Language Foundations for No-Reference Image Quality Assessment](https://mlsanity.com/paper/2509.17374/) (2025)
- [IQA-Adapter: Exploring Knowledge Transfer from Image Quality Assessment to Diffusion-based Generative Models](https://mlsanity.com/paper/2412.01794/) (2024)

## Abstract (arXiv)

The design of image and video quality assessment (QA) algorithms is extremely important to benchmark and calibrate user experience in modern visual systems. A major drawback of the state-of-the-art QA methods is their limited ability to generalize across diverse image and video datasets with reasonable distribution shifts. In this work, we leverage the denoising process of diffusion models for generalized image QA (IQA) and video QA (VQA) by understanding the degree of alignment between learnable quality-aware text prompts and images or video frames. In particular, we learn cross-attention maps from intermediate layers of the denoiser of latent diffusion models (LDMs) to capture quality-aware representations of images or video frames. Since applying text-to-image LDMs for every video frame is computationally expensive for videos, we only estimate the quality of a frame-rate sub-sampled version of the original video. To compensate for the loss in motion information due to frame-rate sub-sampling, we propose a novel temporal quality modulator. Our extensive cross-database experiments across various user-generated, synthetic, low-light, frame-rate variation, ultra high definition, and streaming content-based databases show that our model can achieve superior generalization in both IQA and VQA.