mlsanityData to 7 Oct 2026

DesignProbe: A Graphic Design Benchmark for Multimodal Large Language Models

arXiv 2404.14801 ↗First posted 23 Apr 2024cs.CVcited by 7

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

A well-executed graphic design typically achieves harmony in two levels, from the fine-grained design elements (color, font and layout) to the overall design. This complexity makes the comprehension of graphic design challenging, for it needs the capability to both recognize the design elements and understand the design. With the rapid development of Multimodal Large Language Models (MLLMs), we establish the DesignProbe, a benchmark to investigate the capability of MLLMs in design. Our benchmark includes eight tasks in total, across both the fine-grained element level and the overall design level. At design element level, we consider both the attribute recognition and semantic understanding tasks. At overall design level, we include style and metaphor. 9 MLLMs are tested and we apply GPT-4 as evaluator. Besides, further experiments indicates that refining prompts can enhance the performance of MLLMs. We first rewrite the prompts by different LLMs and found increased performances appear in those who self-refined by their own LLMs. We then add extra task knowledge in two different ways (text descriptions and image examples), finding that adding images boost much more performance over texts.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 43 references that are in the map. First posted on arXiv 23 Apr 2024.

Builds on

37 of 43 references.

References · 37

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
25 more
  1. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  3. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  4. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  5. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  6. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  7. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  8. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  9. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  10. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  11. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  12. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  13. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  14. 2018nocaps: novel object captioning at scalecited by 444
  15. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  16. 2022Visual Prompting via Image Inpaintingcited by 248
  17. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 168
  18. 2023Emu: Generative Pretraining in Multimodalitycited by 156
  19. 2023LayoutDM: Discrete Diffusion Model for Controllable Layout Generationcited by 141
  20. 2021CanvasVAE: Learning to Generate Vector Graphic Documentscited by 93
  21. 2018Coloring with Words: Guiding Image Colorization Through Text-based Palette Generationcited by 35
  22. 2020Predicting Visual Importance Across Graphic Design Typescited by 30
  23. 2023DEsignBench: Exploring and Benchmarking DALL-E 3 for Imagining Visual Designcited by 17
  24. 2022Color Recommendation for Vector Graphic Documents based on Multi-Palette Representationcited by 9
  25. 2023A Survey for Graphic Design Intelligencecited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 3.

7 papers.

  1. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 81
  2. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  3. 2025PosterGen: Aesthetic-Aware Multi-Modal Paper-to-Poster Generation via Multi-Agent LLMscited by 8
  4. 2026Can Vision Language Models Assess Graphic Design Aesthetics? A Benchmark, Evaluation, and Dataset Perspectivecited by 6
  5. 2026PosterIQ: A Design Perspective Benchmark for Poster Understanding and Generationcited by 1
  6. 2025RECALL-MM: A Multimodal Dataset of Consumer Product Recalls for Risk Analysis using Computational Methods and Large Language Modelscited by 0
  7. 2026AUV-Bench: Aesthetic Understanding and Generation Evaluation for User Interfacescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Graphic-Design-Bench: A Comprehensive Benchmark for Evaluating AI on Graphic Design Tasks
  2. 2024From Elements to Design: A Layered Approach for Automatic Graphic Design Composition
  3. 2025Evaluating Graphical Perception with Multimodal LLMs
  4. 2026Mind the Couch! Eliciting MLLM Reasoning in Interior Design via Weak-to-Strong Task Vector Injection
  5. 2024DesignQA: A Multimodal Benchmark for Evaluating Large Language Models' Understanding of Engineering Documentation
  6. 2024Multimodal Markup Document Models for Graphic Design Completion
  7. 2025GraphicBench: A Planning Benchmark for Graphic Design with Language Agents
  8. 2024Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering