mlsanityData to 7 Oct 2026

FILIP: Fine-grained Interactive Language-Image Pre-Training

arXiv 2111.07783 ↗First posted 9 Nov 2021cs.CV cs.LGcited by 626

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Unsupervised large-scale vision-language pre-training has shown promising advances on various downstream tasks. Existing methods often model the cross-modal interaction either via the similarity of the global feature of each modality which misses sufficient information, or finer-grained interactions using cross/self-attention upon visual and textual tokens. However, cross/self-attention suffers from inferior efficiency in both training and inference. In this paper, we introduce a large-scale Fine-grained Interactive Language-Image Pre-training (FILIP) to achieve finer-level alignment through a cross-modal late interaction mechanism, which uses a token-wise maximum similarity between visual and textual tokens to guide the contrastive objective. FILIP successfully leverages the finer-grained expressiveness between image patches and textual words by modifying only contrastive loss, while simultaneously gaining the ability to pre-compute image and text representations offline at inference, keeping both large-scale training and inference efficient. Furthermore, we construct a new large-scale image-text pair dataset called FILIP300M for pre-training. Experiments show that FILIP achieves state-of-the-art performance on multiple downstream vision-language tasks including zero-shot image classification and image-text retrieval. The visualization on word-patch alignment further shows that FILIP can learn meaningful fine-grained features with promising localization ability.

Key facts. Cited by 626 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 55 references that are in the map. First posted on arXiv 9 Nov 2021.

Builds on

42 of 55 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug
  2. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning7 Apr

Other references · 40

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  8. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  9. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  10. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,276
28 more
  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  3. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  4. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  5. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  6. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  7. 2019Contrastive Multiview Codingcited by 1,297
  8. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  9. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  10. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  11. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  12. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  13. 2017Mixed Precision Trainingcited by 1,135
  14. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  15. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  16. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  17. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  18. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  19. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  20. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  21. 2018Stacked Cross Attention for Image-Text Matchingcited by 414
  22. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  23. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  24. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  25. 2020ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Datacited by 136
  26. 2021M6: A Chinese Multimodal Pretrainercited by 91
  27. 2021Product1M: Towards Weakly Supervised Instance-Level Product Retrieval via Cross-modal Pretrainingcited by 42
  28. 2015APAC: Augmented PAttern Classification with Neural Networkscited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 99; 2023: 190; 2024: 150; 2025: 108; 2026: 76.

626 papers.

Sort:

By area:

  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  4. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  5. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  6. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  7. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  8. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  9. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  10. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  11. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  12. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
38 more
  1. 2022Multimodal Learning with Transformers: A Surveycited by 399
  2. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 309
  3. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 279
  4. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  5. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  6. 2023Improving CLIP Training with Language Rewritescited by 216
  7. 2022Delving into Out-of-Distribution Detection with Vision-Language Representationscited by 215
  8. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 215
  9. 2023Learning to Exploit Temporal Structure for Biomedical Vision-Language Processingcited by 210
  10. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  11. 2024ColPali: Efficient Document Retrieval with Vision Language Modelscited by 190
  12. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretrainingcited by 187
  13. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 185
  14. 2022DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detectioncited by 176
  15. 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 162
  16. 2022CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-trainingcited by 159
  17. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  18. 2023RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensingcited by 150
  19. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  20. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  21. 2023Hyperbolic Image-Text Representationscited by 132
  22. 2024Exploring scalable medical image encoders beyond text supervisioncited by 129
  23. 2023MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adaptercited by 128
  24. 2022Unsupervised Prompt Learning for Vision-Language Modelscited by 125
  25. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Modelscited by 118
  26. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  27. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 117
  28. 2022Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmarkcited by 114
  29. 2024ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inferencecited by 112
  30. 2024ProxyCLIP: Proxy Attention Improves CLIP for Open-Vocabulary Segmentationcited by 110
  31. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbonecited by 109
  32. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  33. 2022Chinese CLIP: Contrastive Vision-Language Pretraining in Chinesecited by 108
  34. 2022PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretrainingcited by 104
  35. 2024Low-Rank Few-Shot Adaptation of Vision-Language Modelscited by 103
  36. 2023Align Your Prompts: Test-Time Prompting with Distribution Alignment for Zero-Shot Generalizationcited by 100
  37. 2023Learning Open-vocabulary Semantic Segmentation Models From Natural Language Supervisioncited by 99
  38. 2022Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIPcited by 97

50 most cited of 626 All 626 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025TULIP: Towards Unified Language-Image Pretraining
  2. 2025HarmoCLIP: Harmonizing Global and Regional Representations in Contrastive Vision-Language Models
  3. 2022UniCLIP: Unified Framework for Contrastive Language-Image Pre-training
  4. 2025Explaining Similarity in Vision-Language Encoders with Weighted Banzhaf Interactions
  5. 2025uCLIP: Parameter-Efficient Multilingual Extension of Vision-Language Models with Unpaired Data
  6. 2025FG-CLIP: Fine-Grained Visual and Textual Alignment
  7. 2021SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels
  8. 2025PixCLIP: Achieving Fine-grained Visual Language Understanding via Any-granularity Pixel-Text Alignment Learning