mlsanityData to 14 Oct 2026

Graphic Design with Large Multimodal Model

arXiv 2404.14368 ↗First posted 22 Apr 2024cs.CV cs.AI cs.CLcited by 40

Not on the map. Why?

Abstract (arXiv)

In the field of graphic design, automating the integration of design elements into a cohesive multi-layered artwork not only boosts productivity but also paves the way for the democratization of graphic design. One existing practice is Graphic Layout Generation (GLG), which aims to layout sequential design elements. It has been constrained by the necessity for a predefined correct sequence of layers, thus limiting creative potential and increasing user workload. In this paper, we present Hierarchical Layout Generation (HLG) as a more flexible and pragmatic setup, which creates graphic composition from unordered sets of design elements. To tackle the HLG task, we introduce Graphist, the first layout generation model based on large multimodal models. Graphist efficiently reframes the HLG as a sequence generation problem, utilizing RGB-A images as input, outputs a JSON draft protocol, indicating the coordinates, size, and order of each element. We develop new evaluation metrics for HLG. Graphist outperforms prior arts and establishes a strong baseline for this field. Project homepage: https://github.com/graphic-design-ai/graphist

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Builds on 81 of its 104 references that are in the map. First posted on arXiv 22 Apr 2024.

Builds on

81 of 104 references.

References · 81

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  8. 2023Visual Instruction Tuningcited by 7,869
  9. 2020End-to-End Object Detection with Transformerscited by 7,414
  10. 2020Scaling Laws for Neural Language Modelscited by 6,586
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  12. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  2. 2023Qwen Technical Reportcited by 3,344
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  4. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  7. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  8. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  9. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  10. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  11. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  12. 2023A Survey on Multimodal Large Language Modelscited by 909
  13. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  14. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  15. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  16. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  17. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  18. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  19. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  20. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  21. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  22. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 415
  23. 2019PubLayNet: largest dataset ever for document layout analysiscited by 351
  24. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 349
  25. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 347
  26. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 293
  27. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 290
  28. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 279
  29. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  30. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  31. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 217
  32. 2023Video Understanding with Large Language Models: A Surveycited by 206
  33. 2023Towards Open Vocabulary Learning: A Surveycited by 166
  34. 2020DeepSVG: A Hierarchical Generative Network for Vector Graphics Animationcited by 159
  35. 2023Osprey: Pixel Understanding with Visual Instruction Tuningcited by 159
  36. 2023Emu: Generative Pretraining in Multimodalitycited by 157
  37. 2023LayoutDM: Discrete Diffusion Model for Controllable Layout Generationcited by 143
  38. 2019LayoutGAN: Generating Graphic Layouts with Wireframe Discriminatorscited by 131

50 most cited of 81 All 81 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 19; 2026: 16.

40 papers.

Sort:

By area:

  1. 2024Chat2SVG: Vector Graphics Generation with Large Language Models and Image Diffusion Modelscited by 38
  2. 2025ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generationcited by 37
  3. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  4. 2025CreatiDesign: A Unified Multi-Conditional Diffusion Transformer for Creative Graphic Designcited by 36
  5. 2025BizGen: Advancing Article-level Visual Text Rendering for Infographics Generationcited by 20
  6. 2025CreatiPoster: Towards Editable and Controllable Multi-Layer Graphic Design Generationcited by 19
  7. 2024Multimodal Markup Document Models for Graphic Design Completioncited by 16
  8. 2025PrismLayers: Open Data for High-Quality Multi-Layer Transparent Image Generative Modelscited by 15
  9. 2024Can GPTs Evaluate Graphic Design Based on Design Principles?cited by 14
  10. 2024From Elements to Design: A Layered Approach for Automatic Graphic Design Compositioncited by 13
  11. 2025OmniPSD: Layered PSD Generation with Diffusion Transformercited by 13
  12. 2025BannerAgency: Advertising Banner Design with Multimodal LLM Agentscited by 11
28 more
  1. 2025PosterGen: Aesthetic-Aware Multi-Modal Paper-to-Poster Generation via Multi-Agent LLMscited by 11
  2. 2025Rethinking Layered Graphic Design Generation with a Top-Down Approachcited by 10
  3. 2025LayerPeeler: Autoregressive Peeling for Layer-wise Image Vectorizationcited by 8
  4. 2025IGD: Instructional Graphic Design with Multimodal Layer Generationcited by 8
  5. 2025SEGA: A Stepwise Evolution Paradigm for Content-Aware Layout Generation with Design Priorcited by 6
  6. 2025PosterCopilot: Toward Layout Reasoning and Controllable Editing for Professional Graphic Designcited by 6
  7. 2024Design-o-meter: Towards Evaluating and Refining Graphic Designscited by 5
  8. 2026DesignAsCode: Bridging Structural Editability and Visual Fidelity in Graphic Design Generationcited by 5
  9. 2025GraphicBench: A Planning Benchmark for Graphic Design with Language Agentscited by 4
  10. 2026Seeing is Improving: Visual Feedback for Iterative Text Layout Refinementcited by 4
  11. 2026PaperX: A Unified Framework for Multimodal Academic Presentation Generation with Scholar DAGcited by 3
  12. 2026PSDesigner: Automated Graphic Design with a Human-Like Creative Workflowcited by 2
  13. 2025SheetDesigner: MLLM-Powered Spreadsheet Layout Generation with Rule-Based and Vision-Based Reflectioncited by 1
  14. 2026AnyDoc: Enhancing Document Generation via Large-Scale HTML/CSS Data Synthesis and Height-Aware Reinforcement Optimizationcited by 1
  15. 2026MRT: Masked Region Transformer for Layered Image Generation and Editing at Scalecited by 1
  16. 2026ETPDesigner: Multi-Agent Orchestration for Interactive Multimodal Electronic Theater Programcited by 1
  17. 2025Aggregated Structural Representation with Large Language Models for Human-Centric Layout Generationcited by 0
  18. 2025Sketch-to-Layout: Sketch-Guided Multimodal Layout Generationcited by 0
  19. 2025Step-by-step Layered Design Generationcited by 0
  20. 2026PRISM: Learning Design Knowledge from Data for Stylistic Design Improvementcited by 0
  21. 2026ReLayout: Versatile and Structure-Preserving Design Layout Editing via Relation-Aware Design Reconstructioncited by 0
  22. 2026CPT: Controllable and Editable Design Variations with Language Modelscited by 0
  23. 2026Towards Design Compositingcited by 0
  24. 2026CommerceVibe: Learning to Design E-Commerce Creatives as Executable Visual Code via Dual-Feedback Reinforcement Learningcited by 0
  25. 2026GDB-Reward: From Evaluation Metrics to Training Rewards for Graphic Designcited by 0
  26. 2026Editable Visual Designcited by 0
  27. 2026Learning Interaction between Image and Layout Priors for Joint Image-Layout Generation in Design Templatescited by 0
  28. 2026PLAYGROUND: Progressive Layout Generation with Render-Grounded Planningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024PosterLLaVa: Constructing a Unified Multi-modal Layout Generator with LLM
  2. 2023Unifying Layout Generation with a Decoupled Diffusion Model
  3. 2024LayoutDiT: Exploring Content-Graphic Balance in Layout Generation with Diffusion Transformer
  4. 2026LICA: Layered Image Composition Annotations for Graphic Design Research
  5. 2023COLE: A Hierarchical Generation Framework for Multi-Layered and Editable Graphic Design
  6. 2026ArtChart: Faithful Artistic Chart Generation with Integrated Text Rendering
  7. 2024Towards Aligned Layout Generation via Diffusion Model with Aesthetic Constraints
  8. 2025ComposeAnyone: Controllable Layout-to-Human Generation with Decoupled Multimodal Conditions