Exploring Diagnostic Prompting Approach for Multimodal LLM-based Visual Complexity Assessment: A Case Study of Amazon Search Result Pages
On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning
Abstract (arXiv)
This study investigates whether diagnostic prompting can improve Multimodal Large Language Model (MLLM) reliability for visual complexity assessment of Amazon Search Results Pages (SRP). We compare diagnostic prompting with standard gestalt principles-based prompting using 200 Amazon SRP pages and human expert annotations. Diagnostic prompting showed notable improvements in predicting human complexity judgments, with F1-score increasing from 0.031 to 0.297 (+858% relative improvement), though absolute performance remains modest (Cohen's kappa = 0.071). The decision tree revealed that models prioritize visual design elements (badge clutter: 38.6% importance) while humans emphasize content similarity, suggesting partial alignment in reasoning patterns. Failure case analysis reveals persistent challenges in MLLM visual perception, particularly for product similarity and color intensity assessment. Our findings indicate that diagnostic prompting represents a promising initial step toward human-aligned MLLM-based evaluation, though failure cases with consistent human-MLLM disagreement require continued research and refinement in prompting approaches with larger ground truth datasets for reliable practical deployment.
Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 13 of its 39 references that are in the map. First posted on arXiv 26 Nov 2025.
Builds on
13 of 39 references.
Closest ancestors · 1
Same direction, newest first
Other references · 12
- 2023Visual Instruction Tuningcited by 7,827
- 2023A survey of the Vision Transformers and their CNN-Transformer based Variantscited by 33
- 2025HueManity: Probing Fine-Grained Visual Perception in MLLMscited by 9
- 2025Interpretable and Reliable Detection of AI-Generated Images via Grounded Reasoning in MLLMscited by 9
- 2025VER-Bench: Evaluating MLLMs on Reasoning with Fine-Grained Visual Evidencecited by 9
- 2024A Comparative Survey of Vision Transformers for Feature Extraction in Texture Analysiscited by 6
- 2025Visually Interpretable Subtask Reasoning for Visual Question Answeringcited by 4
- 2024The Gestalt Computational Model by Persistent Homologycited by 3
- 2024Finding Closure: A Closer Look at the Gestalt Law of Closure in Convolutional Neural Networkscited by 3
- 2024POV Learning: Individual Alignment of Multimodal Models using Human Perceptioncited by 1
- 2025Cognitive Science-Inspired Evaluation of Core Capabilities for Object Understanding in AIcited by 1
- 2025Multimodal LLM Augmented Reasoning for Interpretable Visual Perception Analysiscited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Human-Like Attention? A Psychophysical Comparison of Visual Search in Humans and MLLMs
- 2025Evaluating Graphical Perception with Multimodal LLMs
- 2024A Comprehensive Study of Multimodal Large Language Models for Image Quality Assessment
- 2025MRAG-Suite: A Diagnostic Evaluation Platform for Visual Retrieval-Augmented Generation
- 2026When Prompts Mislead: Textual Dominance and Diagnostic Bias in MLLMs
- 2025Towards Synthesizing Normative Data for Cognitive Assessments Using Generative Multimodal Large Language Models
- 2025Do You See Me : A Multidimensional Benchmark for Evaluating Visual Perception in Multimodal LLMs
- 2023VRPTEST: Evaluating Visual Referring Prompting in Large Multimodal Models