mlsanityData to 7 Oct 2026

Let ViT Speak: Generative Language-Image Pre-training

arXiv 2605.00809 ↗First posted 1 May 2026cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

In this paper, we present Generative Language-Image Pre-training (GenLIP), a minimalist generative pretraining framework for Vision Transformers (ViTs) designed for multimodal large language models (MLLMs). To better align vision encoders with the autoregressive nature of LLMs, GenLIP trains a ViT to predict language tokens directly from visual tokens using a standard language modeling objective, without contrastive batch construction or an additional text decoder. This design offers three key advantages: (1) Simplicity: a single transformer jointly models visual and textual tokens; (2) Scalability: it scales effectively with both data and model size; and (3) Performance: it achieves competitive or superior results across diverse multimodal benchmarks. Trained on 8B samples from Recap-DataComp-1B, GenLIP matches or surpasses strong baselines despite using substantially less pretraining data. After continued pretraining on multi-resolution images at native aspect ratios, GenLIP further improves on detail-sensitive tasks such as OCR and chart understanding, making it a strong foundation for vision encoders in MLLMs.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 81 of its 92 references that are in the map. First posted on arXiv 1 May 2026.

Builds on

81 of 92 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025From Pixels to Words -- Towards Native Vision-Language Primitives at Scale16 Oct
  2. 2025CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learning26 Sep
  3. 2025DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real World30 Jun
  4. 2025Kimi-VL Technical Report10 Apr
  5. 2025EVEv2: Improved Baselines for Encoder-Free Vision-Language Models10 Feb
  6. 2024PaliGemma: A versatile 3B VLM for transfer10 Jul
  7. 2024SOLO: A Single Transformer for Scalable Vision-Language Modeling8 Jul
  8. 2024Unveiling Encoder-Free Vision-Language Models17 Jun
  9. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  10. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug
  11. 2023Revisiting the Role of Language Priors in Vision-Language Models2 Jun
  12. 2022VL-BEiT: Generative Vision-Language Pretraining2 Jun
3 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan
  3. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug

Other references · 66

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2025Qwen3 Technical Reportcited by 8,195
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  9. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2023Qwen Technical Reportcited by 3,273
  12. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
38 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  4. 2025Qwen3-VL Technical Reportcited by 2,534
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  9. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  10. 2019Towards VQA Models That Can Readcited by 1,696
  11. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  12. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  13. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  14. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  15. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  16. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  17. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
  18. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  19. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  20. 2023Vision Transformers Need Registerscited by 844
  21. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  22. 2016A Diagram Is Worth A Dozen Imagescited by 793
  23. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  24. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  25. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
  26. 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
  27. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  28. 2018nocaps: novel object captioning at scalecited by 444
  29. 2021InfographicVQAcited by 430
  30. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  31. 2025BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Datasetcited by 363
  32. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 344
  33. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  34. 2023Demystifying CLIP Datacited by 265
  35. 2023Improving CLIP Training with Language Rewritescited by 216
  36. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehensioncited by 144
  37. 2024Multimodal Autoregressive Pre-training of Large Vision Encoderscited by 108
  38. 2023Image Captioners Are Scalable Vision Learners Toocited by 89

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026IronViT: Toward Efficient Generalist Visual Representation Learningcited by 0
  2. 2026How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretrainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Expedited Training of Visual Conditioned Language Generation via Redundancy Reduction
  2. 2023MAGVLT: Masked Generative Vision-and-Language Transformer
  3. 2023MiniGPT-5: Interleaved Vision-and-Language Generation via Generative Vokens
  4. 2025Visual Generation Tuning
  5. 2025Growing Visual Generative Capacity for Pre-Trained MLLMs
  6. 2021Unified Multimodal Pre-training and Prompt-based Tuning for Vision-Language Understanding and Generation
  7. 2023Bootstrapping Vision-Language Learning with Decoupled Language Pre-training
  8. 2025VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Model