mlsanityData to 7 Oct 2026

Towards Flexible Multi-modal Document Models

arXiv 2303.18248 ↗First posted 31 Mar 2023cs.CVcited by 30

Not on the map. Why?

Abstract (arXiv)

Creative workflows for generating graphical documents involve complex inter-related tasks, such as aligning elements, choosing appropriate fonts, or employing aesthetically harmonious colors. In this work, we attempt at building a holistic model that can jointly solve many different design tasks. Our model, which we denote by FlexDM, treats vector graphic documents as a set of multi-modal elements, and learns to predict masked fields such as element type, position, styling attributes, image, or text, using a unified architecture. Through the use of explicit multi-task learning and in-domain pre-training, our model can better capture the multi-modal relationships among the different document fields. Experimental results corroborate that our single FlexDM is able to successfully solve a multitude of different design tasks, while achieving performance that is competitive with task-specific and costly baselines.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 37 of its 58 references that are in the map. First posted on arXiv 31 Mar 2023.

Builds on

37 of 58 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  7. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  8. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  9. 2017Learning multiple visual domains with residual adapterscited by 553
  10. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 549
  11. 2018End-to-End Multi-Task Learning with Attentioncited by 533
  12. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
25 more
  1. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 406
  2. 2017A Neural Representation of Sketch Drawingscited by 311
  3. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  4. 2016UberNet: Training a `Universal' Convolutional Neural Network for Low-, Mid-, and High-Level Vision using Diverse Datasets and Limited Memorycited by 200
  5. 2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
  6. 2018Efficient parametrization of multi-domain deep neural networkscited by 168
  7. 2020DeepSVG: A Hierarchical Generative Network for Vector Graphics Animationcited by 158
  8. 2023LayoutDM: Discrete Diffusion Model for Controllable Layout Generationcited by 141
  9. 2019LayoutGAN: Generating Graphic Layouts with Wireframe Discriminatorscited by 130
  10. 2020LayoutTransformer: Layout Generation and Completion with Self-attentioncited by 122
  11. 2019LayoutVAE: Stochastic Scene Layout Generation From a Label Setcited by 119
  12. 2021SelfDoc: Self-Supervised Document Representation Learningcited by 117
  13. 2021UIBert: Learning Generic Multimodal Representations for UI Understandingcited by 93
  14. 2021CanvasVAE: Learning to Generate Vector Graphic Documentscited by 93
  15. 2019A Learned Representation for Scalable Vector Graphicscited by 91
  16. 2021Variational Transformer Networks for Layout Generationcited by 88
  17. 2019Neural Design Network: Graphic Layout Generation with Constraintscited by 83
  18. 2021Constrained Graphic Layout Generation via Latent Optimizationcited by 83
  19. 2021BLT: Bidirectional Layout Transformer for Controllable Layout Generationcited by 69
  20. 2021DOC2PPT: Automatic Presentation Slides Generation from Scientific Documentscited by 68
  21. 2020Attribute-conditioned Layout GAN for Automatic Graphic Designcited by 67
  22. 2020ActionBert: Leveraging User Actions for Semantic Understanding of User Interfacescited by 43
  23. 2021InfoColorizer: Interactive Recommendation of Color Palettes for Infographicscited by 15
  24. 2022Color Recommendation for Vector Graphic Documents based on Multi-Palette Representationcited by 9
  25. 2022Generative Colorization of Structured Mobile Web Pagescited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 8; 2025: 14; 2026: 5.

30 papers.

Sort:

By area:

  1. 2023COLE: A Hierarchical Generation Framework for Multi-Layered and Editable Graphic Designcited by 39
  2. 2024Graphic Design with Large Multimodal Modelcited by 38
  3. 2025ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generationcited by 37
  4. 2024OpenCOLE: Towards Reproducible Automatic Graphic Design Generationcited by 36
  5. 2025CreatiDesign: A Unified Multi-Conditional Diffusion Transformer for Creative Graphic Designcited by 36
  6. 2025AVD2: Accident Video Diffusion for Accident Video Descriptioncited by 20
  7. 2025CreatiPoster: Towards Editable and Controllable Multi-Layer Graphic Design Generationcited by 17
  8. 2023Planning and Rendering: Towards Product Poster Generation with Diffusion Modelscited by 15
  9. 2024Multimodal Markup Document Models for Graphic Design Completioncited by 15
  10. 2024From Elements to Design: A Layered Approach for Automatic Graphic Design Compositioncited by 11
  11. 2025BannerAgency: Advertising Banner Design with Multimodal LLM Agentscited by 11
  12. 2025Rethinking Layered Graphic Design Generation with a Top-Down Approachcited by 9
18 more
  1. 2025AesthetiQ: Enhancing Graphic Layout Design via Aesthetic-Aware Preference Alignment of Multi-modal Large Language Modelscited by 8
  2. 2024Automatic Layout Planning for Visually-Rich Documents with Instruction-Following Modelscited by 7
  3. 2023FASTER: A Font-Agnostic Scene Text Editing and Rendering Frameworkcited by 6
  4. 2025SEGA: A Stepwise Evolution Paradigm for Content-Aware Layout Generation with Design Priorcited by 6
  5. 2024DocSynthv2: A Practical Autoregressive Modeling for Document Generationcited by 5
  6. 2024TextCenGen: Attention-Guided Text-Centric Background Adaptation for Text-to-Image Generationcited by 3
  7. 2024CoLay: Controllable Layout Generation through Multi-conditional Latent Diffusioncited by 3
  8. 2025Agentic Design Review Systemcited by 2
  9. 2026Seeing is Improving: Visual Feedback for Iterative Text Layout Refinementcited by 2
  10. 2025DesignLab: Designing Slides Through Iterative Detection and Correctioncited by 1
  11. 2026AnyDoc: Enhancing Document Generation via Large-Scale HTML/CSS Data Synthesis and Height-Aware Reinforcement Optimizationcited by 1
  12. 2026PSDesigner: Automated Graphic Design with a Human-Like Creative Workflowcited by 1
  13. 2025Generating Animated Layouts as Structured Text Representationscited by 0
  14. 2025Automatic Text Box Placement for Supporting Typographic Designcited by 0
  15. 2025Sketch-to-Layout: Sketch-Guided Multimodal Layout Generationcited by 0
  16. 2025Step-by-step Layered Design Generationcited by 0
  17. 2026ReLayout: Versatile and Structure-Preserving Design Layout Editing via Relation-Aware Design Reconstructioncited by 0
  18. 2026DesignSense: A Human Preference Dataset and Reward Modeling Framework for Graphic Layout Generationcited by 0