Can MLLMs Critique Like Humans? Evaluating Open-Ended Aesthetic Reasoning in Multimodal Large Language Models
On the map: LLM reasoning → Multimodal reasoning
Abstract (arXiv)
Open-ended aesthetic critique is a challenge for multimodal large language models (MLLMs): it has no single correct answer, and most aesthetic evaluation measures models against numeric scores rather than the written critiques people actually give. We ask whether MLLM critiques are close to human ones, scoring eight open-weight MLLMs from 7B to 397B, plus GPT-5.5, against multiple ranked human critiques for each of 1,227 textttr/photocritique posts under eight prompt conditions. Reference-based similarity gives a misleading picture. In absolute terms the stricter lexical and learned metrics align only weakly with human critiques while a coarse embedding cosine reports broad topical overlap, yet requesting shorter critiques raises those scores and withholding the image barely changes them: the similarity reflects length, the post text, and a stable critiquing style more than image-specific observation. An LLM judge sharpens the question rather than settling it: in the primary condition all four judges prefer the frontier models' critiques to the human ones, but on the 7--8B models they diverge wildly, from 9% to 81% preference on identical pairs. Asked instead how similar each pair is in substance, those judges and two human annotators agree, rating every model between 1.81 and 2.59 on a 1--5 scale, close to ``mostly different''. Behaviorally, the models diverge in ways the scores do not surface: they cover nearly every aesthetic aspect where humans are selective and repeat themselves across critiques of one photo, even when prompted to write at human length. We argue that reference-based similarity rewards a fluent, comprehensive critique style rather than the selectivity and specificity of human critique.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 34 of its 50 references that are in the map. First posted on arXiv 29 Jun 2026.
Builds on
34 of 50 references.
References · 34
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
- 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
- 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
22 more
- 2023Large Language Models are not Fair Evaluatorscited by 1,085
- 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
- 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
- 2023Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levelscited by 503
- 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 418
- 2017NIMA: Neural Image Assessmentcited by 389
- 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 249
- 2017Why We Need New Evaluation Metrics for NLGcited by 233
- 2020Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metricscited by 158
- 2016Photo Aesthetics Ranking Network with Attributes and Content Adaptationcited by 125
- 2024AesBench: An Expert Benchmark for Multimodal Large Language Models on Image Aesthetics Perceptioncited by 65
- 2024UNIAA: A Unified Multi-modal Image Aesthetic Assessment Baseline and Benchmarkcited by 42
- 2022Understanding Aesthetics with Language: A Photo Critique Dataset for Aesthetic Assessmentcited by 15
- 2025The Photographer Eye: Teaching Multimodal Large Language Models to Understand Image Aesthetics like Photographerscited by 13
- 2019Aesthetic Image Captioning From Weakly-Labelled Photographscited by 12
- 2025AesBiasBench: Evaluating Bias and Alignment in Multimodal Language Models for Personalized Image Aesthetic Assessmentcited by 11
- 2025Few-Shot Adversarial Low-Rank Fine-Tuning of Vision-Language Modelscited by 9
- 2025VISTA: Vision-Language Inference for Training-Free Stock Time-Series Analysiscited by 6
- 2026MMLoP: Multi-Modal Low-Rank Prompting for Efficient Vision-Language Adaptationcited by 5
- 2025pFedMMA: Personalized Federated Fine-Tuning with Multi-Modal Adapter for Vision-Language Modelscited by 4
- 2025From Filters to VLMs: Benchmarking Defogging Methods through Object Detection and Segmentation Performancecited by 3
- 2026NeuroMambaLLM: Dynamic Graph Learning of fMRI Functional Connectivity in Autistic Brains Using Mamba and Language Model Reasoningcited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026AesCanvas: A Large-Scale Dataset and Benchmark for Aesthetic Critique and Contextual Suitability
- 2024AesExpert: Towards Multi-modality Foundation Model for Image Aesthetics Perception
- 2025MM-CRITIC: A Holistic Evaluation of Large Multimodal Models as Multimodal Critique
- 2026Visual Aesthetic Benchmark: Can Frontier Models Judge Beauty?
- 2024II-Bench: An Image Implication Understanding Benchmark for Multimodal Large Language Models
- 2025AesTest: Measuring Aesthetic Intelligence from Perception to Production
- 2025LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models
- 2025Same Content, Different Answers: Cross-Modal Inconsistency in MLLMs