# IQAGPT: Image Quality Assessment with Vision-language and ChatGPT Models

arXiv: https://arxiv.org/abs/2312.15663 · HTML: https://mlsanity.com/paper/2312.15663/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 63 references that are in the map.
- First posted on arXiv 25 Dec 2023.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Exploring CLIP for Assessing the Look and Feel of Images](https://mlsanity.com/paper/2207.12396/) (2022)
- [Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models](https://mlsanity.com/paper/2303.04671/) (2023)
- [Low-Dose CT with a Residual Encoder-Decoder Convolutional Neural Network (RED-CNN)](https://mlsanity.com/paper/1702.00288/) (2017)
- [Low Dose CT Image Denoising Using a Generative Adversarial Network with Wasserstein Distance and Perceptual Loss](https://mlsanity.com/paper/1708.00961/) (2017)
- [Can Deep Learning Outperform Modern Commercial CT Image Reconstruction Methods?](https://mlsanity.com/paper/1811.03691/) (2018)
- [CoreDiff: Contextual Error-Modulated Generalized Diffusion Model for Low-Dose CT Denoising and Generalization](https://mlsanity.com/paper/2304.01814/) (2023)
- [3D Convolutional Encoder-Decoder Network for Low-Dose CT via Transfer Learning from a 2D Trained Network](https://mlsanity.com/paper/1802.05656/) (2018)
- [Translating Radiology Reports into Plain Language using ChatGPT and GPT-4 with Prompt Learning: Promising Results, Limitations, and Potential](https://mlsanity.com/paper/2303.09038/) (2023)
- [CONVIQT: Contrastive Video Quality Estimator](https://mlsanity.com/paper/2206.14713/) (2022)
- [ASCON: Anatomy-aware Supervised Contrastive Learning Framework for Low-dose CT Denoising](https://mlsanity.com/paper/2307.12225/) (2023)
- [LIT-Former: Linking In-plane and Through-plane Transformers for Simultaneous CT Image Denoising and Deblurring](https://mlsanity.com/paper/2302.10630/) (2023)
- [Self-supervised Multi-modal Training from Uncurated Image and Reports Enables Zero-shot Oversight Artificial Intelligence in Radiology](https://mlsanity.com/paper/2208.05140/) (2022)

## Built on by (most cited first, 18 of 18)

- [LMM-VQA: Advancing Video Quality Assessment with Large Multimodal Models](https://mlsanity.com/paper/2408.14008/) (2024)
- [AGHI-QA: A Subjective-Aligned Dataset and Metric for AI-Generated Human Images](https://mlsanity.com/paper/2504.21308/) (2025)
- [Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Review](https://mlsanity.com/paper/2506.18378/) (2025)
- [Low-dose CT Denoising with Language-engaged Dual-space Alignment](https://mlsanity.com/paper/2403.06128/) (2024)
- [Understanding Generative AI Capabilities in Everyday Image Editing Tasks](https://mlsanity.com/paper/2505.16181/) (2025)
- [Radiologist-in-the-Loop Self-Training for Generalizable CT Metal Artifact Reduction](https://mlsanity.com/paper/2501.15610/) (2025)
- [LangMamba: A Language-driven Mamba Framework for Low-dose CT Denoising with Vision-language Models](https://mlsanity.com/paper/2507.06140/) (2025)
- [Autoadaptive Medical Segment Anything Model](https://mlsanity.com/paper/2507.01828/) (2025)
- [FoundDiff: Foundational Diffusion Model for Generalizable Low-Dose CT Denoising](https://mlsanity.com/paper/2508.17299/) (2025)
- [HiRQA: Hierarchical Ranking and Quality Alignment for Opinion-Unaware Image Quality Assessment](https://mlsanity.com/paper/2508.15130/) (2025)
- [Contrast-Prior Enhanced Duality for Mask-Free Shadow Removal](https://mlsanity.com/paper/2507.21949/) (2025)
- [LMM-IQA: Image Quality Assessment for Low-Dose CT Imaging](https://mlsanity.com/paper/2511.07298/) (2025)
- [Prompt-Conditioned FiLM and Multi-Scale Fusion on MedSigLIP for Low-Dose CT Quality Assessment](https://mlsanity.com/paper/2511.12256/) (2025)
- [CAP-IQA: Context-Aware Prompt-Guided CT Image Quality Assessment](https://mlsanity.com/paper/2601.01613/) (2026)
- [R4-CGQA: Retrieval-based Vision Language Models for Computer Graphics Image Quality Assessment](https://mlsanity.com/paper/2603.10578/) (2026)
- [MedQ-UNI: Toward Unified Medical Image Quality Assessment and Restoration via Vision-Language Modeling](https://mlsanity.com/paper/2603.18465/) (2026)
- [MedQ-Engine: A Closed-Loop Data Engine for Evolving MLLMs in Medical Image Quality Assessment](https://mlsanity.com/paper/2603.19863/) (2026)
- [Harnessing Vision-Language Models for Perceptual Quality Assessment and Autonomous Content Adjustment in Augmented Reality](https://mlsanity.com/paper/2610.00677/) (2026)

## Abstract (arXiv)

Large language models (LLMs), such as ChatGPT, have demonstrated impressive capabilities in various tasks and attracted an increasing interest as a natural language interface across many domains. Recently, large vision-language models (VLMs) like BLIP-2 and GPT-4 have been intensively investigated, which learn rich vision-language correlation from image-text pairs. However, despite these developments, the application of LLMs and VLMs in image quality assessment (IQA), particularly in medical imaging, remains to be explored, which is valuable for objective performance evaluation and potential supplement or even replacement of radiologists' opinions. To this end, this paper introduces IQAGPT, an innovative image quality assessment system integrating an image quality captioning VLM with ChatGPT for generating quality scores and textual reports. First, we build a CT-IQA dataset for training and evaluation, comprising 1,000 CT slices with diverse quality levels professionally annotated. To better leverage the capabilities of LLMs, we convert annotated quality scores into semantically rich text descriptions using a prompt template. Second, we fine-tune the image quality captioning VLM on the CT-IQA dataset to generate quality descriptions. The captioning model fuses the image and text features through cross-modal attention. Third, based on the quality descriptions, users can talk with ChatGPT to rate image quality scores or produce a radiological quality report. Our preliminary results demonstrate the feasibility of assessing image quality with large models. Remarkably, our IQAGPT outperforms GPT-4 and CLIP-IQA, as well as the multi-task classification and regression models that solely rely on images.