Data to 7 Oct 2026

VILA: Learning Image Aesthetics from User Comments with Vision-Language Pretraining

arXiv 2303.14302 ↗First posted 24 Mar 2023cs.CVcited by 63

On the map: Multimodal LLMs / vision-language models → Image editing and quality assessment

Abstract (arXiv)

Assessing the aesthetics of an image is challenging, as it is influenced by multiple factors including composition, color, style, and high-level semantics. Existing image aesthetic assessment (IAA) methods primarily rely on human-labeled rating scores, which oversimplify the visual aesthetic information that humans perceive. Conversely, user comments offer more comprehensive information and are a more natural way to express human opinions and preferences regarding image aesthetics. In light of this, we propose learning image aesthetics from user comments, and exploring vision-language pretraining methods to learn multimodal aesthetic representations. Specifically, we pretrain an image-text encoder-decoder model with image-comment pairs, using contrastive and generative objectives to learn rich and generic aesthetic semantics without human labels. To efficiently adapt the pretrained model for downstream IAA tasks, we further propose a lightweight rank-based adapter that employs text as an anchor to learn the aesthetic ranking concept. Our results show that our pretrained aesthetic vision-language model outperforms prior works on image aesthetic captioning over the AVA-Captions dataset, and it has powerful zero-shot capability for aesthetic tasks such as zero-shot style classification and zero-shot IAA, surpassing many supervised baselines. With only minimal finetuning parameters using the proposed adapter module, our model achieves state-of-the-art IAA performance over the AVA dataset.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Placed in Image editing and quality assessment, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 73 references that are in the map. First posted on arXiv 24 Mar 2023.

Builds on

38 of 73 references.

References · 38

2017Attention Is All You Needcited by 59,932
2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
2016Overcoming catastrophic forgetting in neural networkscited by 4,718
2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224

26 more

2021MUSIQ: Multi-scale Image Quality Transformercited by 1,058
2021Scaling Vision Transformerscited by 993
2022Exploring CLIP for Assessing the Look and Feel of Imagescited by 974
2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
2022MaxViT: Multi-Axis Vision Transformercited by 476
2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
2017NIMA: Neural Image Assessmentcited by 387
2019KonIQ-10k: An ecologically valid database for deep learning of blind image quality assessmentcited by 351
2019Blind Image Quality Assessment Using A Deep Bilinear Convolutional Neural Networkcited by 301
2019UNITER: UNiversal Image-TExt Representation Learningcited by 225
2016Deep Neural Networks for No-Reference and Full-Reference Image Quality Assessmentcited by 224
2019From Patches to Pictures (PaQ-2-PiQ): Mapping the Perceptual Space of Picture Qualitycited by 221
2014Unifying Visual-Semantic Embeddings with Multimodal Neural Language Modelscited by 197
2020MetaIQA: Deep Meta-learning for No-Reference Image Quality Assessmentcited by 144
2016Photo Aesthetics Ranking Network with Attributes and Content Adaptationcited by 122
2013Recognizing Image Stylecited by 64
2022Personalized Image Aesthetics Assessment with Rich Attributescited by 56
2019Effective Aesthetics Prediction with Multi-level Spatially Pooled Featurescited by 52
2017A Probabilistic Quality Representation Approach to Deep Blind Image Quality Predictioncited by 28
2020Adaptive Fractional Dilated Convolution Network for Image Aesthetics Assessmentcited by 28
2019Aesthetic Image Captioning From Weakly-Labelled Photographscited by 12
2018Neural Aesthetic Image Reviewercited by 11
2017A deep architecture for unified aesthetic predictioncited by 6
2022Image Aesthetics Assessment Using Graph Attention Networkcited by 5
2019A Geometry-Sensitive Approach for Photographic Style Classificationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 22; 2025: 23; 2026: 10.

63 papers.

Sort:

By area:

2023Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levelscited by 489
2023Q-Instruct: Improving Low-level Visual Abilities for Multi-modality Foundation Modelscited by 152
2023DreamSync: Aligning Text-to-Image Generation with Image Understanding Feedbackcited by 57
2023Towards Explainable In-the-Wild Video Quality Assessment: A Database and a Language-Prompted Approachcited by 45
2024Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedbackcited by 45
2024PRDP: Proximal Reward Difference Prediction for Large-Scale Reward Finetuning of Diffusion Modelscited by 41
2025Calibrated Multi-Preference Optimization for Aligning Diffusion Modelscited by 40
2024Q-Bench-Video: Benchmarking the Video Quality Understanding of LMMscited by 39
2024Parrot: Pareto-optimal Multi-Reward Reinforcement Learning Framework for Text-to-Image Generationcited by 35
2024Quality-Aware Image-Text Alignment for Opinion-Unaware Image Quality Assessmentcited by 31
2024Quality Assessment in the Era of Large Models: A Surveycited by 26
2023SPIRE: Semantic Prompt-Driven Image Restorationcited by 24

38 more

2024NTIRE 2024 Quality Assessment of AI-Generated Content Challengecited by 20
2025HumanAesExpert: Advancing a Multi-Modality Foundation Model for Human Image Aesthetic Assessmentcited by 17
2024VQA2: Visual Question Answering for Video Quality Assessmentcited by 15
2024CLIP-AGIQA: Boosting the Performance of AI-Generated Image Quality Assessment with CLIPcited by 14
2024UniQA: Unified Vision-Language Pre-training for Image Quality and Aesthetic Assessmentcited by 12
2024Unsupervised Domain Adaption Harnessing Vision-Language Pre-trainingcited by 12
2025Next Token Is Enough: Realistic Image Quality and Aesthetic Scoring with Multimodal Large Language Modelcited by 11
2024Advancing Text-Driven Chest X-Ray Generation with Policy-Based Reinforcement Learningcited by 10
2024Cropper: Vision-Language Model for Image Cropping through In-Context Learningcited by 10
2025SAGI: Semantically Aligned and Uncertainty Guided AI Image Inpaintingcited by 10
2025How far can we go with ImageNet for Text-to-Image generation?cited by 10
2023Long-Term Ad Memorability: Understanding & Generating Memorable Adscited by 9
2024AGFSync: Leveraging AI-Generated Feedback for Preference Optimization in Text-to-Image Generationcited by 9
2024QPT V2: Masked Image Modeling Advances Visual Scoringcited by 8
2024Scaling Up Personalized Image Aesthetic Assessment via Task Vector Customizationcited by 6
2024KnobGen: Controlling the Sophistication of Artwork in Sketch-Based Diffusion Modelscited by 6
2025Exploring Cognitive and Aesthetic Causality for Multimodal Aspect-Based Sentiment Analysiscited by 6
2025Diffusion Blend: Inference-Time Multi-Preference Alignment for Diffusion Modelscited by 6
2024ATTIQA: Generalizable Image Quality Feature Extractor using Attribute-aware Pretrainingcited by 5
2025Language-Guided Visual Perception Disentanglement for Image Quality Assessment and Conditional Image Generationcited by 5
2025Gamma: Toward Generic Image Assessment with Mixture of Assessment Expertscited by 5
2025Charm: The Missing Piece in ViT fine-tuning for Image Aesthetic Assessmentcited by 5
2025UniRes: Universal Image Restoration for Complex Degradationscited by 5
2025Unlocking the Essence of Beauty: Advanced Aesthetic Reasoning with Relative-Absolute Policy Optimizationcited by 5
2026Fine-grained Image Aesthetic Assessment: Learning Discriminative Scores from Relative Rankscited by 5
2023CLIP Brings Better Features to Visual Aesthetics Learnerscited by 4
2023Image Aesthetics Assessment via Learnable Queriescited by 4
2024AIS 2024 Challenge on Video Quality Assessment of User-Generated Content: Methods and Resultscited by 4
2024Commonly Interesting Imagescited by 3
2024Ranking-aware adapter for text-driven image ordering with CLIPcited by 2
2025Q-Router: Agentic Video Quality Assessment with Expert Model Routing and Artifact Localizationcited by 2
2025MapReduce LoRA: Advancing the Pareto Front in Multi-Preference Optimization for Generative Modelscited by 2
2026Personalized Image Generation via Human-in-the-loop Bayesian Optimizationcited by 2
2025On the Role of Individual Differences in Current Approaches to Computational Image Aestheticscited by 1
2025Iterative Self-Improvement of Vision Language Models for Image Scoring and Self-Explanationcited by 1
2025OmniQuality-R: Advancing Reward Models Through All-Encompassing Quality Assessmentcited by 1
2025Visual Interestingness Decoded: How GPT-4o Mirrors Human Interestscited by 1
2025AesTest: Measuring Aesthetic Intelligence from Perception to Productioncited by 1

50 most cited of 63 All 63 (JSON)

Similar, not cited

Close in wording, no citation either way