mlsanityData to 7 Oct 2026

Augmenting Vision Language Pretraining by Learning Codebook with Visual Semantics

arXiv 2208.00475 ↗First posted 31 Jul 2022cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Language modality within the vision language pretraining framework is innately discretized, endowing each word in the language vocabulary a semantic meaning. In contrast, visual modality is inherently continuous and high-dimensional, which potentially prohibits the alignment as well as fusion between vision and language modalities. We therefore propose to "discretize" the visual representation by joint learning a codebook that imbues each visual token a semantic. We then utilize these discretized visual semantics as self-supervised ground-truths for building our Masked Image Modeling objective, a counterpart of Masked Language Modeling which proves successful for language models. To optimize the codebook, we extend the formulation of VQ-VAE which gives a theoretic guarantee. Experiments validate the effectiveness of our approach across common vision-language benchmarks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 45 references that are in the map. First posted on arXiv 31 Jul 2022.

Builds on

38 of 45 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Vision-Language Pre-Training with Triple Contrastive Learning21 Feb
  2. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning7 Apr

Other references · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  9. 2017Neural Discrete Representation Learningcited by 4,650
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
24 more
  1. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  3. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  5. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  6. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  7. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  8. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  9. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  10. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  11. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  12. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  13. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  14. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  15. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  16. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  17. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  18. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  19. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  20. 2021PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformerscited by 204
  21. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  22. 2020ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Datacited by 136
  23. 2022Multi-modal Alignment using Representation Codebookcited by 63
  24. 2020SLADE: A Self-Training Framework For Distance Metric Learningcited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Leveraging per Image-Token Consistency for Vision-Language Pre-training
  2. 2022VL-BEiT: Generative Vision-Language Pretraining
  3. 2022Efficient Vision-Language Pretraining with Visual Concepts and Hierarchical Alignment
  4. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Models
  5. 2025Context Matters: Learning Global Semantics via Object-Centric Representation
  6. 2026LeVLJEPA: End-to-End Vision-Language Pretraining Without Negatives
  7. 2022MVPTR: Multi-Level Semantic Alignment for Vision-Language Pre-Training via Multi-Stage Learning
  8. 2025From Pixels to Words -- Towards Native Vision-Language Primitives at Scale