mlsanityData to 14 Oct 2026

DEsignBench: Exploring and Benchmarking DALL-E 3 for Imagining Visual Design

arXiv 2310.15144 ↗First posted 23 Oct 2023cs.CVcited by 17

Not on the map. Why?

Abstract (arXiv)

We introduce DEsignBench, a text-to-image (T2I) generation benchmark tailored for visual design scenarios. Recent T2I models like DALL-E 3 and others, have demonstrated remarkable capabilities in generating photorealistic images that align closely with textual inputs. While the allure of creating visually captivating images is undeniable, our emphasis extends beyond mere aesthetic pleasure. We aim to investigate the potential of using these powerful models in authentic design contexts. In pursuit of this goal, we develop DEsignBench, which incorporates test samples designed to assess T2I models on both "design technical capability" and "design application scenario." Each of these two dimensions is supported by a diverse set of specific design categories. We explore DALL-E 3 together with other leading T2I models on DEsignBench, resulting in a comprehensive visual gallery for side-by-side comparisons. For DEsignBench benchmarking, we perform human evaluations on generated images in DEsignBench gallery, against the criteria of image-text alignment, visual aesthetic, and design creativity. Our evaluation also considers other specialized design capabilities, including text rendering, layout composition, color harmony, 3D design, and medium style. In addition to human evaluations, we introduce the first automatic image generation evaluator powered by GPT-4V. This evaluator provides ratings that align well with human judgments, while being easily replicable and cost-efficient. A high-resolution version is available at https://github.com/design-bench/design-bench.github.io/raw/main/designbench.pdf?download=

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 50 of its 101 references that are in the map. First posted on arXiv 23 Oct 2023.

Builds on

50 of 101 references.

References · 50

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  8. 2017Neural Discrete Representation Learningcited by 4,712
  9. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  11. 2013Auto-Encoding Variational Bayescited by 4,053
  12. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
38 more
  1. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  2. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  3. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  4. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,043
  5. 2023Training Diffusion Models with Reinforcement Learningcited by 982
  6. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  7. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  8. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  9. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 702
  10. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  11. 2017Unsupervised Image-to-Image Translation Networkscited by 603
  12. 2015A Neural Algorithm of Artistic Stylecited by 483
  13. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 450
  14. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 445
  15. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 422
  16. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesiscited by 388
  17. 2023GPTScore: Evaluate as You Desirecited by 361
  18. 2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 345
  19. 2023T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generationcited by 260
  20. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  21. 2023Break-A-Scene: Extracting Multiple Concepts from a Single Imagecited by 231
  22. 2023TextDiffuser: Diffusion Models as Text Painterscited by 229
  23. 2023Subject-driven Text-to-Image Generation via Apprenticeship Learningcited by 212
  24. 2023Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speechcited by 166
  25. 2023GlyphControl: Glyph Conditional Control for Visual Text Generationcited by 131
  26. 2023Wider and Deeper LLM Networks are Fairer LLM Evaluatorscited by 94
  27. 2022Character-Aware Models Improve Visual Text Renderingcited by 92
  28. 2023Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?cited by 88
  29. 2022A very preliminary analysis of DALL-E 2cited by 82
  30. 2023ImagenHub: Standardizing the evaluation of conditional image generation modelscited by 77
  31. 2023Visual Programming for Text-to-Image Generation and Evaluationcited by 54
  32. 2023Calibrating LLM-Based Evaluatorcited by 53
  33. 2023Idea2Img: Iterative Self-Refinement with GPT-4V(ision) for Automatic Image Design and Generationcited by 38
  34. 2023GameEval: Evaluating LLMs on Conversational Gamescited by 37
  35. 2023GlyphDraw: Seamlessly Rendering Text with Intricate Spatial Structures in Text-to-Image Generationcited by 34
  36. 2023A Closer Look into Automatic Evaluation Using Large Language Modelscited by 17
  37. 2023OpenLEAF: Open-Domain Interleaved Image-Text Generation and Evaluationcited by 16
  38. 2023Towards Diverse and Consistent Typography Generationcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 7; 2026: 4.

17 papers.

Sort:

By area:

  1. 2024Multi-LoRA Composition for Image Generationcited by 80
  2. 2023COLE: A Hierarchical Generation Framework for Multi-Layered and Editable Graphic Designcited by 43
  3. 2025BizGen: Advancing Article-level Visual Text Rendering for Infographics Generationcited by 20
  4. 2025SridBench: Benchmark of Scientific Research Illustration Drawing of Image Generation Modelcited by 19
  5. 2024VisualCritic: Making LMMs Perceive Visual Quality Like Humanscited by 13
  6. 2024DesignProbe: A Graphic Design Benchmark for Multimodal Large Language Modelscited by 7
  7. 2024Finding the Subjective Truth: Collecting 2 Million Votes for Comprehensive Gen-AI Model Evaluationcited by 7
  8. 2024IDEA-Bench: How Far are Generative Models from Professional Designing?cited by 7
  9. 2026Can Vision Language Models Assess Graphic Design Aesthetics? A Benchmark, Evaluation, and Dataset Perspectivecited by 6
  10. 2025The Moon's Many Faces: A Single Unified Transformer for Multimodal Lunar Reconstructioncited by 3
  11. 2025LoRAtorio: An intrinsic approach to LoRA Skill Compositioncited by 3
  12. 2026CodePercept: Code-Grounded Visual STEM Perception for MLLMscited by 3
5 more
  1. 2025Replace in Translation: Boost Concept Alignment in Counterfactual Text-to-Imagecited by 1
  2. 2026HistDiT: A Structure-Aware Latent Conditional Diffusion Model for High-Fidelity Virtual Staining in Histopathologycited by 1
  3. 2025Dynamic Relation Inference via Verb Embeddingscited by 0
  4. 2025MEF: A Systematic Evaluation Framework for Text-to-Image Modelscited by 0
  5. 2026Training-Free Multi-Concept Image Editingcited by 0