mlsanityData to 7 Oct 2026

Efficient Vision-Language Pretraining with Visual Concepts and Hierarchical Alignment

arXiv 2208.13628 ↗First posted 29 Aug 2022cs.CV cs.LGcited by 19

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision and Language Pretraining has become the prevalent approach for tackling multimodal downstream tasks. The current trend is to move towards ever larger models and pretraining datasets. This computational headlong rush does not seem reasonable in the long term to move toward sustainable solutions, and de facto excludes academic laboratories with limited resources. In this work, we propose a new framework, dubbed ViCHA, that efficiently exploits the input data to boost the learning by: (a) a new hierarchical cross-modal alignment loss, (b) new self-supervised scheme based on masked image modeling, (c) leveraging image-level annotations, called Visual Concepts, obtained with existing foundation models such as CLIP to boost the performance of the image encoder. Although pretrained on four times less data, our ViCHA strategy outperforms other approaches on several downstream tasks such as Image-Text Retrieval, VQA, Visual Reasoning, Visual Entailment and Visual Grounding. The code will be made publicly available here: https://github.com/mshukor/ViCHA

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 98 of its 111 references that are in the map. First posted on arXiv 29 Aug 2022.

Builds on

98 of 111 references.

Closest ancestors · 12

Same direction, newest first

  1. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone15 Jun
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  3. 2022Vision-Language Pre-Training with Triple Contrastive Learning21 Feb
  4. 2022A Survey of Vision-Language Pre-Trained Models18 Feb
  5. 2022VLP: A Survey on Vision-Language Pre-training18 Feb
  6. 2021Scaling Up Vision-Language Pre-training for Image Captioning24 Nov
  7. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts16 Nov
  8. 2021FILIP: Fine-grained Interactive Language-Image Pre-Training9 Nov
  9. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts3 Nov
  10. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug
  11. 2021Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training25 Jun
  12. 2021SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels14 Mar

Other references · 86

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  11. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  12. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
38 more
  1. 2020Scaling Laws for Neural Language Modelscited by 6,411
  2. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  6. 2015VQA: Visual Question Answeringcited by 2,489
  7. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  8. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  9. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  11. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  12. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  13. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  14. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  15. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  16. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  17. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  18. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  19. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  20. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  21. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  22. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  23. 2016Modeling Context in Referring Expressionscited by 1,054
  24. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  25. 2021Scaling Vision Transformerscited by 993
  26. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  27. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  28. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  29. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  30. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  31. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  32. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  33. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  34. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  35. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  36. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  37. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  38. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405

50 most cited of 86 All 86 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 12; 2024: 5; 2025: 1.

19 papers.

Sort:

By area:

  1. 2023Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learningcited by 53
  2. 2023UnIVAL: Unified Model for Image, Video, Audio and Language Taskscited by 44
  3. 2023eP-ALM: Efficient Perceptual Augmentation of Language Modelscited by 33
  4. 2023Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learningcited by 30
  5. 2023LIP-Loc: LiDAR Image Pretraining for Cross-Modal Localizationcited by 26
  6. 2024Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputscited by 22
  7. 2023Improved baselines for vision-language pre-trainingcited by 20
  8. 2024Improved Baselines for Data-efficient Perceptual Augmentation of LLMscited by 18
  9. 2023Video-Teller: Enhancing Cross-Modal Generation with Fusion and Decouplingcited by 12
  10. 2023GIVL: Improving Geographical Inclusivity of Vision-Language Models with Pre-Training Methodscited by 11
  11. 2023Bi-VLGM : Bi-Level Class-Severity-Aware Vision-Language Graph Matching for Text Guided Medical Image Segmentationcited by 10
  12. 20233VL: Using Trees to Improve Vision-Language Models' Interpretabilitycited by 7
7 more
  1. 2024Skipping Computations in Multimodal LLMscited by 7
  2. 2023BDC-Adapter: Brownian Distance Covariance for Better Vision-Language Reasoningcited by 6
  3. 2024Optimizing CLIP Models for Image Retrieval with Maintained Joint-Embedding Alignmentcited by 6
  4. 2024From Orthogonality to Dependency: Learning Disentangled Representation for Multi-Modal Time-Series Sensing Signalscited by 4
  5. 2022Vision and Structured-Language Pretraining for Cross-Modal Food Retrievalcited by 3
  6. 2023Aerial Image Object Detection With Vision Transformer Detector (ViTDet)cited by 3
  7. 2025Image Embedding Sampling Method for Diverse Captioningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages
  2. 2022Augmenting Vision Language Pretraining by Learning Codebook with Visual Semantics
  3. 2021Unified Multimodal Pre-training and Prompt-based Tuning for Vision-Language Understanding and Generation
  4. 2025Visual Representation Alignment for Multimodal Large Language Models
  5. 2022X2-VLM: All-In-One Pre-trained Model For Vision-Language Tasks
  6. 2023VILA: On Pre-training for Visual Language Models
  7. 2022VL-BEiT: Generative Vision-Language Pretraining
  8. 2023ViLTA: Enhancing Vision-Language Pre-training through Textual Augmentation