# SEAGULL: No-reference Image Quality Assessment for Regions of Interest via Vision-Language Instruction Tuning

arXiv: https://arxiv.org/abs/2411.10161 · HTML: https://mlsanity.com/paper/2411.10161/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Image editing and quality assessment, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 40 of its 68 references that are in the map.
- First posted on arXiv 15 Nov 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Image editing and quality assessment](https://mlsanity.com/mllm/image-editing-and-quality-assessment/)

## Builds on (most cited first, 30 of 40)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [MUSIQ: Multi-scale Image Quality Transformer](https://mlsanity.com/paper/2108.05997/) (2021)
- [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023)
- [mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration](https://mlsanity.com/paper/2311.04257/) (2023)
- [MANIQA: Multi-dimension Attention Network for No-Reference Image Quality Assessment](https://mlsanity.com/paper/2204.08958/) (2022)
- [Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levels](https://mlsanity.com/paper/2312.17090/) (2023)
- [Segment Anything in High Quality](https://mlsanity.com/paper/2306.01567/) (2023)
- [KonIQ-10k: An ecologically valid database for deep learning of blind image quality assessment](https://mlsanity.com/paper/1910.06180/) (2019)
- [Blind Image Quality Assessment Using A Deep Bilinear Convolutional Neural Network](https://mlsanity.com/paper/1907.02665/) (2019)
- [Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspective](https://mlsanity.com/paper/2303.14968/) (2023)
- [Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision](https://mlsanity.com/paper/2309.14181/) (2023)
- [TOPIQ: A Top-down Approach from Semantics to Distortions for Image Quality Assessment](https://mlsanity.com/paper/2308.03060/) (2023)
- [From Patches to Pictures (PaQ-2-PiQ): Mapping the Perceptual Space of Picture Quality](https://mlsanity.com/paper/1912.10088/) (2019)
- [Massive Online Crowdsourced Study of Subjective and Objective Picture Quality](https://mlsanity.com/paper/1511.02919/) (2015)
- [No-Reference Image Quality Assessment via Transformers, Relative Ranking, and Self-Consistency](https://mlsanity.com/paper/2108.06858/) (2021)
- [Osprey: Pixel Understanding with Visual Instruction Tuning](https://mlsanity.com/paper/2312.10032/) (2023)
- [Q-Instruct: Improving Low-level Visual Abilities for Multi-modality Foundation Models](https://mlsanity.com/paper/2311.06783/) (2023)
- [RankIQA: Learning from Rankings for No-reference Image Quality Assessment](https://mlsanity.com/paper/1707.08347/) (2017)
- [Depicting Beyond Scores: Advancing Image Quality Assessment through Multi-modal Language Models](https://mlsanity.com/paper/2312.08962/) (2023)
- [Blind Quality Assessment for in-the-Wild Images via Hierarchical Feature Fusion and Iterative Mixed Database Training](https://mlsanity.com/paper/2105.14550/) (2021)
- [Norm-in-Norm Loss with Faster Convergence and Better Performance for Image Quality Assessment](https://mlsanity.com/paper/2008.03889/) (2020)
- [Hallucinated-IQA: No-Reference Image Quality Assessment via Adversarial Learning](https://mlsanity.com/paper/1804.01681/) (2018)

## Built on by (most cited first, 13 of 13)

- [Q-Ponder: A Unified Training Pipeline for Reasoning-based Visual Quality Assessment](https://mlsanity.com/paper/2506.05384/) (2025)
- [Next Token Is Enough: Realistic Image Quality and Aesthetic Scoring with Multimodal Large Language Model](https://mlsanity.com/paper/2503.06141/) (2025)
- [NTIRE 2025 challenge on Text to Image Generation Model Quality Assessment](https://mlsanity.com/paper/2505.16314/) (2025)
- [Revisiting MLLM Based Image Quality Assessment: Errors and Remedy](https://mlsanity.com/paper/2511.07812/) (2025)
- [Investigate the Low-level Visual Perception in Vision-Language based Image Quality Assessment](https://mlsanity.com/paper/2512.09573/) (2025)
- [FinPercep-RM: A Fine-grained Reward Model and Co-evolutionary Curriculum for RL-based Real-world Super-Resolution](https://mlsanity.com/paper/2512.22647/) (2025)
- [ViDA-UGC: Detailed Image Quality Analysis via Visual Distortion Assessment for UGC Images](https://mlsanity.com/paper/2508.12605/) (2025)
- [AgenticIQA: An Agentic Framework for Adaptive and Interpretable Image Quality Assessment](https://mlsanity.com/paper/2509.26006/) (2025)
- [Building Reasonable Inference for Vision-Language Models in Blind Image Quality Assessment](https://mlsanity.com/paper/2512.09555/) (2025)
- [Decoupling Perception and Calibration: Label-Efficient Image Quality Assessment Framework](https://mlsanity.com/paper/2601.20689/) (2026)
- [Panoptic Pairwise Distortion Graph](https://mlsanity.com/paper/2604.11004/) (2026)
- [IPAD-CLIP: Teaching CLIP to Detect Image Local Perceptual Artifacts](https://mlsanity.com/paper/2605.08664/) (2026)
- [From Glance to Scrutiny: Progressive Distortion Reasoning for Fine-Grained Image Quality Assessment](https://mlsanity.com/paper/2609.08316/) (2026)

## Abstract (arXiv)

Existing Image Quality Assessment (IQA) methods achieve remarkable success in analyzing quality for overall image, but few works explore quality analysis for Regions of Interest (ROIs). The quality analysis of ROIs can provide fine-grained guidance for image quality improvement and is crucial for scenarios focusing on region-level quality. This paper proposes a novel network, SEAGULL, which can SEe and Assess ROIs quality with GUidance from a Large vision-Language model. SEAGULL incorporates a vision-language model (VLM), masks generated by Segment Anything Model (SAM) to specify ROIs, and a meticulously designed Mask-based Feature Extractor (MFE) to extract global and local tokens for specified ROIs, enabling accurate fine-grained IQA for ROIs. Moreover, this paper constructs two ROI-based IQA datasets, SEAGULL-100w and SEAGULL-3k, for training and evaluating ROI-based IQA. SEAGULL-100w comprises about 100w synthetic distortion images with 33 million ROIs for pre-training to improve the model's ability of regional quality perception, and SEAGULL-3k contains about 3k authentic distortion ROIs to enhance the model's ability to perceive real world distortions. After pre-training on SEAGULL-100w and fine-tuning on SEAGULL-3k, SEAGULL shows remarkable performance on fine-grained ROI quality assessment. Code and datasets are publicly available at the https://github.com/chencn2020/Seagull.