mlsanityData to 14 Oct 2026

Beyond Atomic Layouts: Compositional Design Understanding with Vision-Language Models

arXiv 2608.26716 ↗First posted 27 Aug 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Layout understanding, or the interpretation of element organization, is essential for document analysis, user interface (UI) creation, and graphic design. While recent vision-language models (VLMs) excel at interpreting atomic layouts composed of independent elements, they struggle with compositional layouts that require reasoning over visually entangled elements within hierarchical multi-layer structures. In this paper, we introduce a new task, compositional layout understanding, and present CoDeLayout, a VQA dataset of ~20K real-world multi-layer layouts annotated with compositional element pairs and design intent. Through empirical analysis on CoDeLayout, we identify two key challenges for existing VLMs: semantic drift between textual metadata and visual content, and structural ambiguity in hierarchical inter-element relationships. To address these challenges, we propose MASON, a post-training paradigm that integrates multimodal alignment (MA) and structural perception (SP). MA enhances element interpretation by grounding metadata-defined elements to their visual counterparts, mitigating semantic drift, while SP models layer-aware inter-element spatial relationships to improve hierarchical understanding and reduce structural ambiguity. Experiments reveal substantial gaps in existing VLMs: even the strongest baseline, GPT-o3, achieves only 79.68% accuracy, whereas Qwen2.5-VL 7B with MASON reaches 91.66%. Notably, MASON surpasses full-data Direct Finetune using only 30% of the training data and scales better with additional data.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 56 references that are in the map. First posted on arXiv 27 Aug 2026.

Builds on

43 of 56 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025AesthetiQ: Enhancing Graphic Layout Design via Aesthetic-Aware Preference Alignment of Multi-modal Large Language Models1 Mar

Other references · 42

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2025Qwen2.5-VL Technical Reportcited by 5,587
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  10. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  11. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  12. 2023Qwen Technical Reportcited by 3,344
30 more
  1. 2025Qwen3-VL Technical Reportcited by 2,793
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  3. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  4. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  5. 2023A Survey on Multimodal Large Language Modelscited by 909
  6. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  7. 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
  8. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 553
  9. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 485
  10. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 408
  11. 2019PubLayNet: largest dataset ever for document layout analysiscited by 351
  12. 2022DiT: Self-supervised Pre-training for Document Image Transformercited by 152
  13. 2024Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineeringcited by 152
  14. 2023LayoutDM: Discrete Diffusion Model for Controllable Layout Generationcited by 143
  15. 2019LayoutGAN: Generating Graphic Layouts with Wireframe Discriminatorscited by 131
  16. 2020LayoutTransformer: Layout Generation and Completion with Self-attentioncited by 123
  17. 2019LayoutVAE: Stochastic Scene Layout Generation From a Label Setcited by 119
  18. 2017pix2code: Generating Code from a Graphical User Interface Screenshotcited by 114
  19. 2021Constrained Graphic Layout Generation via Latent Optimizationcited by 83
  20. 2022ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshotscited by 66
  21. 2025ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generationcited by 37
  22. 2025LayerD: Decomposing Raster Graphic Designs into Layerscited by 21
  23. 2026ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Modelcited by 13
  24. 2025VQToken: Neural Discrete Token Representation Learning for Extreme Token Reduction in Video Large Language Modelscited by 12
  25. 2025Representation Potentials of Foundation Models for Multimodal Alignment: A Surveycited by 11
  26. 2025SEGA: A Stepwise Evolution Paradigm for Content-Aware Layout Generation with Design Priorcited by 6
  27. 2023Text2Poster: Laying out Stylized Texts on Retrieved Imagescited by 5
  28. 2024Benchmarking VLMs' Reasoning About Persuasive Atypical Imagescited by 5
  29. 2026The Indra Representation Hypothesis for Multimodal Alignmentcited by 4
  30. 2025D-CoDe: Scaling Image-Pretrained VLMs to Video via Dynamic Compression and Question Decompositioncited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026DOne: Decoupling Structure and Rendering for High-Fidelity Design-to-Code Generation
  2. 2025ReLayout: Integrating Relation Reasoning for Content-aware Layout Generation with Multi-modal Large Language Models
  3. 2026DocCogito: Aligning Layout Cognition and Step-Level Grounded Reasoning for Document Understanding
  4. 2025PPTBench: Towards Holistic Evaluation of Large Language Models for PowerPoint Layout and Design Understanding
  5. 2025CAL-RAG: Retrieval-Augmented Multi-Agent Generation for Content-Aware Layout Design
  6. 2025Aggregated Structural Representation with Large Language Models for Human-Centric Layout Generation
  7. 2026From Pixels to Policies: Reinforcing Spatial Reasoning in Language Models for Content-Aware Layout Design
  8. 2024PosterLLaVa: Constructing a Unified Multi-modal Layout Generator with LLM