mlsanityData to 7 Oct 2026

Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding

arXiv 2210.03347 ↗First posted 7 Oct 2022cs.CL cs.CVcited by 360

Not on the map. Why?

Cited by 118 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Visually-situated language is ubiquitous -- sources range from textbooks with diagrams to web pages with images and tables, to mobile apps with buttons and forms. Perhaps due to this diversity, previous work has typically relied on domain-specific recipes with limited sharing of the underlying data, model architectures, and objectives. We present Pix2Struct, a pretrained image-to-text model for purely visual language understanding, which can be finetuned on tasks containing visually-situated language. Pix2Struct is pretrained by learning to parse masked screenshots of web pages into simplified HTML. The web, with its richness of visual elements cleanly reflected in the HTML structure, provides a large source of pretraining data well suited to the diversity of downstream tasks. Intuitively, this objective subsumes common pretraining signals such as OCR, language modeling, image captioning. In addition to the novel pretraining strategy, we introduce a variable-resolution input representation and a more flexible integration of language and vision inputs, where language prompts such as questions are rendered directly on top of the input image. For the first time, we show that a single pretrained model can achieve state-of-the-art results in six out of nine tasks across four domains: documents, illustrations, user interfaces, and natural images.

Key facts. Cited by 360 papers in the map (data through 7 Oct 2026). Builds on 48 of its 62 references that are in the map. First posted on arXiv 7 Oct 2022.

Builds on

48 of 62 references.

References · 48

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  7. 2022Emergent Abilities of Large Language Modelscited by 2,483
  8. 2019Towards VQA Models That Can Readcited by 1,696
  9. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  10. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  11. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  12. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
36 more
  1. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  2. 2016A Diagram Is Worth A Dozen Imagescited by 793
  3. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  4. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  5. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  6. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 481
  7. 2021InfographicVQAcited by 430
  8. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 406
  9. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  10. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  11. 2021DocFormer: End-to-End Transformer for Document Understandingcited by 254
  12. 2019Fixing the train-test resolution discrepancycited by 209
  13. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 180
  14. 2022A Unified Sequence Interface for Vision Taskscited by 152
  15. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  16. 2021Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformercited by 134
  17. 2021Screen2Words: Automatic Mobile UI Summarization with Multimodal Learningcited by 119
  18. 2020TAP: Text-Aware Pre-training for Text-VQA and Text-Captioncited by 95
  19. 2020Widget Captioning: Generating Natural Language Description for Mobile User Interface Elementscited by 94
  20. 2021UIBert: Learning Generic Multimodal Representations for UI Understandingcited by 93
  21. 2021StructuralLM: Structural Pre-training for Form Understandingcited by 91
  22. 2016Image-to-Markup Generation with Coarse-to-Fine Attentioncited by 79
  23. 2022End-to-end Document Recognition and Understanding with Dessurtcited by 73
  24. 2021LaTr: Layout-Aware Transformer for Scene-Text VQAcited by 72
  25. 2020Understanding tables with intermediate pre-trainingcited by 68
  26. 2022Spotlight: Mobile UI Understanding using Vision-Language Models with a Focuscited by 62
  27. 2021HTLM: Hyper-Text Pre-Training and Prompting of Language Modelscited by 58
  28. 2021Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixelscited by 57
  29. 2022Language Modelling with Pixelscited by 53
  30. 2022Grounding Answers for Visual Questions Asked by Visually Impaired Peoplecited by 47
  31. 2021MarkupLM: Pre-training of Text and Markup Language for Visually-rich Document Understandingcited by 44
  32. 2020ActionBert: Leveraging User Actions for Semantic Understanding of User Interfacescited by 43
  33. 2022WebFormer: The Web-page Transformer for Structure Information Extractioncited by 34
  34. 2021Screen Parsing: Towards Reverse Engineering of UI Models from Screenshotscited by 32
  35. 2021VUT: Versatile UI Transformer for Multi-Modal Multi-Task User Interface Modelingcited by 26
  36. 2020Unblind Your Apps: Predicting Natural-Language Labels for Mobile GUI Components by Deep Learningcited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 60; 2024: 128; 2025: 79; 2026: 88.

360 papers.

Sort:

By area:

  1. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  2. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  3. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  4. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  5. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  6. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 424
  7. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 370
  8. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  9. 2024What matters when building vision-language models?cited by 321
  10. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  11. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  12. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 237
38 more
  1. 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 226
  2. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
  3. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  4. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 177
  5. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 174
  6. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Documentcited by 173
  7. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  8. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  9. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 167
  10. 2025WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learningcited by 162
  11. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 159
  12. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 158
  13. 2023ChartLlama: A Multimodal LLM for Chart Understanding and Generationcited by 158
  14. 2022DePlot: One-shot visual language reasoning by plot-to-table translationcited by 157
  15. 2023MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuningcited by 149
  16. 2024Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agentscited by 144
  17. 2023UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Modelcited by 143
  18. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 137
  19. 2024Aria: An Open Multimodal Native Mixture-of-Experts Modelcited by 136
  20. 2022MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derenderingcited by 132
  21. 2024ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoningcited by 127
  22. 2024Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Datasetcited by 126
  23. 2023PaLI-3 Vision Language Models: Smaller, Faster, Strongercited by 125
  24. 2023ChartBench: A Benchmark for Complex Visual Reasoning in Chartscited by 116
  25. 2024mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understandingcited by 113
  26. 2023DocLLM: A layout-aware generative language model for multimodal document understandingcited by 112
  27. 2024GUICourse: From General Vision Language Models to Versatile GUI Agentscited by 112
  28. 2024M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understandingcited by 110
  29. 2025Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generationcited by 108
  30. 2023DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understandingcited by 106
  31. 2024Best Practices and Lessons Learned on Synthetic Datacited by 104
  32. 2024GUI Agents: A Surveycited by 101
  33. 2024ScreenAI: A Vision-Language Model for UI and Infographics Understandingcited by 97
  34. 2023KOSMOS-2.5: A Multimodal Literate Modelcited by 95
  35. 2024LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understandingcited by 94
  36. 2023Image Captioners Are Scalable Vision Learners Toocited by 89
  37. 2025A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Modelscited by 87
  38. 2023Vary: Scaling up the Vision Vocabulary for Large Vision-Language Modelscited by 84

50 most cited of 360 All 360 (JSON)