mlsanityData to 7 Oct 2026

DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generation

arXiv 2410.00201 ↗First posted 30 Sep 2024cs.CV cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

Enabling machines to understand structured visuals like slides and user interfaces is essential for making them accessible to people with disabilities. However, achieving such understanding computationally has required manual data collection and annotation, which is time-consuming and labor-intensive. To overcome this challenge, we present a method to generate synthetic, structured visuals with target labels using code generation. Our method allows people to create datasets with built-in labels and train models with a small number of human-annotated examples. We demonstrate performance improvements in three tasks for understanding slides and UIs: recognizing visual elements, describing visual content, and classifying visual content types.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 50 of its 86 references that are in the map. First posted on arXiv 30 Sep 2024.

Builds on

50 of 86 references.

References · 50

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  5. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  6. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  7. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  8. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  9. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  10. 2019EfficientDet: Scalable and Efficient Object Detectioncited by 1,018
  11. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  12. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
38 more
  1. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
  2. 2023Synthetic Data from Diffusion Models Improves ImageNet Classificationcited by 326
  3. 2023DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Datacited by 322
  4. 2022Language Models are Realistic Tabular Data Generatorscited by 290
  5. 2023StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learnerscited by 177
  6. 2023SlideVQA: A Dataset for Document Visual Question Answering on Multiple Imagescited by 175
  7. 2023ChartLlama: A Multimodal LLM for Chart Understanding and Generationcited by 158
  8. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 143
  9. 2021Probabilistic two-stage detectioncited by 130
  10. 2018Learning Semantic Segmentation from Synthetic Data: A Geometrically Guided Input-Output Adaptation Approachcited by 119
  11. 2021Screen2Words: Automatic Mobile UI Summarization with Multimodal Learningcited by 119
  12. 2023InPars-v2: Large Language Models as Efficient Dataset Generators for Information Retrievalcited by 73
  13. 2023ChartGPT: Leveraging LLMs to Generate Charts from Abstract Natural Languagecited by 71
  14. 2024Instruct-Imagen: Image Generation with Multi-modal Instructioncited by 65
  15. 2023PosterLayout: A New Benchmark and Approach for Content-aware Visual-Textual Presentation Layoutcited by 60
  16. 2021Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixelscited by 57
  17. 2022GRIT: Faster and Better Image captioning Transformer Using Dual Visual Featurescited by 56
  18. 2023WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semanticscited by 45
  19. 2023Faithful Persona-based Conversational Dataset Generation with Large Language Modelscited by 45
  20. 2023Prompt2Model: Generating Deployable Models from Natural Language Instructionscited by 40
  21. 2021VINS: Visual Search for Mobile User Interface Designcited by 37
  22. 2018Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Appscited by 36
  23. 2024A Vision Check-up for Language Modelscited by 36
  24. 2021Screen Parsing: Towards Reverse Engineering of UI Models from Screenshotscited by 32
  25. 2023Let's Synthesize Step by Step: Iterative Dataset Synthesis with Large Language Models by Extrapolating Errors from Small Modelscited by 30
  26. 2020Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or a Combination?cited by 29
  27. 2024UICoder: Finetuning Large Language Models to Generate User Interface Code through Automated Feedbackcited by 26
  28. 2023AVscript: Accessible Video Editing with Audio-Visual Scriptscited by 19
  29. 2019Modeling Mobile Interface Tappability Using Crowdsourcing and Deep Learningcited by 18
  30. 2023GenAssist: Making Image Generation Accessiblecited by 18
  31. 2023ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversationscited by 17
  32. 2024UIClip: A Data-driven Model for Assessing User Interface Designcited by 14
  33. 2021Say It All: Feedback for Improving Non-Visual Presentation Accessibilitycited by 9
  34. 2022Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency Analysiscited by 9
  35. 2023Never-ending Learning of User Interfacescited by 9
  36. 2024Keyframer: Empowering Animation Design using Large Language Modelscited by 9
  37. 2016Large-Scale Query-by-Image Video Retrieval Using Bloom Filterscited by 6
  38. 2022Reflow: Automatically Improving Touch Interactions in Mobile Applications through Pixel-based Refinementscited by 2

Built on by

Cited by year

Citing papers per year: 2025: 8.

8 papers.

  1. 2025AutoPresent: Designing Structured Visuals from Scratchcited by 63
  2. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 31
  3. 2025PosterO: Structuring Layout Trees to Enable Language Models in Generalized Content-Aware Layout Generationcited by 15
  4. 2025TEMPURA: Temporal Event Masked Prediction and Understanding for Reasoning in Actioncited by 13
  5. 2025A Large Scale Analysis of Gender Biases in Text-to-Image Generative Modelscited by 12
  6. 2025DesignPref: Capturing Personal Preferences in Visual Design Generationcited by 6
  7. 2025SciPostGen: Bridging the Gap between Scientific Papers and Poster Layoutscited by 4
  8. 2025DesignLab: Designing Slides Through Iterative Detection and Correctioncited by 1