mlsanityData to 7 Oct 2026

Enhancing Vision-Language Pre-training with Rich Supervisions

arXiv 2403.03346 ↗First posted 5 Mar 2024cs.CVcited by 12

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

We propose Strongly Supervised pre-training with ScreenShots (S4) - a novel pre-training paradigm for Vision-Language Models using data from large-scale web screenshot rendering. Using web screenshots unlocks a treasure trove of visual and textual cues that are not present in using image-text pairs. In S4, we leverage the inherent tree-structured hierarchy of HTML elements and the spatial localization to carefully design 10 pre-training tasks with large scale annotated data. These tasks resemble downstream tasks across different domains and the annotations are cheap to obtain. We demonstrate that, compared to current screenshot pre-training objectives, our innovative pre-training method significantly enhances performance of image-to-text model in nine varied and popular downstream tasks - up to 76.1% improvements on Table Detection, and at least 1% on Widget Captioning.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 71 of its 77 references that are in the map. First posted on arXiv 5 Mar 2024.

Builds on

71 of 77 references.

Closest ancestors · 11

Same direction, newest first

  1. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone15 Jun
  2. 2022Vision-Language Pre-Training with Triple Contrastive Learning21 Feb
  3. 2022A Survey of Vision-Language Pre-Trained Models18 Feb
  4. 2022VLP: A Survey on Vision-Language Pre-training18 Feb
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan
  6. 2021Scaling Up Vision-Language Pre-training for Image Captioning24 Nov
  7. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts16 Nov
  8. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts3 Nov
  9. 2021E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning3 Jun
  10. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning7 Apr
  11. 2021Causal Attention for Vision-Language Tasks5 Mar

Other references · 60

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2022Training language models to follow instructions with human feedbackcited by 15,712
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  9. 2023Segment Anythingcited by 8,158
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
38 more
  1. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  3. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  5. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  7. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  8. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  9. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  10. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  11. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  12. 2021Grounded Language-Image Pre-trainingcited by 1,147
  13. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  14. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  15. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  16. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  17. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
  18. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  19. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  20. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  21. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  22. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 379
  23. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  24. 2019PubLayNet: largest dataset ever for document layout analysiscited by 349
  25. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  26. 2021DocFormer: End-to-End Transformer for Document Understandingcited by 254
  27. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Modelscited by 182
  28. 2021PubTables-1M: Towards comprehensive table extraction from unstructured documentscited by 122
  29. 2021Screen2Words: Automatic Mobile UI Summarization with Multimodal Learningcited by 119
  30. 2021VinVL: Revisiting Visual Representations in Vision-Language Modelscited by 109
  31. 2021A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Modelscited by 101
  32. 2021WebSRC: A Dataset for Web-Based Structural Reading Comprehensioncited by 98
  33. 2020Widget Captioning: Generating Natural Language Description for Mobile User Interface Elementscited by 94
  34. 2021UIBert: Learning Generic Multimodal Representations for UI Understandingcited by 93
  35. 2021LaTr: Layout-Aware Transformer for Scene-Text VQAcited by 72
  36. 2021Kaleido-BERT: Vision-Language Pre-training on Fashion Domaincited by 67
  37. 2022Masked Vision and Language Modeling for Multi-modal Representation Learningcited by 66
  38. 2022Spotlight: Mobile UI Understanding using Vision-Language Models with a Focuscited by 62

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 3; 2026: 2.

12 papers.

  1. 2024VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?cited by 98
  2. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  3. 2024Harnessing Webpage UIs for Text-Rich Visual Understandingcited by 28
  4. 2025R-VLM: Region-Aware Vision Language Model for Precise GUI Groundingcited by 20
  5. 2024TinyClick: Single-Turn Agent for Empowering GUI Automationcited by 12
  6. 2024ClickAgent: Enhancing UI Location Capabilities of Autonomous Agentscited by 11
  7. 2024EDGE: Enhanced Grounded GUI Understanding with Enriched Multi-Granularity Synthetic Datacited by 10
  8. 2025Turbocharging Web Automation: The Impact of Compressed History Statescited by 2
  9. 2026Scaling Dense Event-Stream Pretraining from Visual Foundation Modelscited by 2
  10. 2026Towards GUI Agents: Vision-Language Diffusion Models for GUI Groundingcited by 2
  11. 2024MiniGPT-Reverse-Designing: Predicting Image Adjustments Utilizing MiniGPT-4cited by 0
  12. 2025Group-CLIP Uncertainty Modeling for Group Re-Identificationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages
  2. 2026Focusing Where Vision Matters: Selective Training for Large Vision Language Models via Visual Information Gain
  3. 2022Vision-Language Pre-Training for Boosting Scene Text Detectors
  4. 2023CiT: Curation in Training for Effective Vision-Language Data
  5. 2025Scaling Pre-training to One Hundred Billion Data for Vision Language Models
  6. 2023SILC: Improving Vision Language Pretraining with Self-Distillation
  7. 2026InstAP: Instance-Aware Vision-Language Pre-Train for Spatial-Temporal Understanding
  8. 2025Prioritizing Image-Related Tokens Enhances Vision-Language Pre-Training