mlsanityData to 7 Oct 2026

Contrastive Language-Image Pre-Training with Knowledge Graphs

arXiv 2210.08901 ↗First posted 17 Oct 2022cs.CVcited by 34

Not on the map. Why?

Abstract (arXiv)

Recent years have witnessed the fast development of large-scale pre-training frameworks that can extract multi-modal representations in a unified form and achieve promising performances when transferred to downstream tasks. Nevertheless, existing approaches mainly focus on pre-training with simple image-text pairs, while neglecting the semantic connections between concepts from different modalities. In this paper, we propose a knowledge-based pre-training framework, dubbed Knowledge-CLIP, which injects semantic information into the widely used CLIP model. Through introducing knowledge-based objectives in the pre-training process and utilizing different types of knowledge graphs as training data, our model can semantically align the representations in vision and language with higher quality, and enhance the reasoning ability across scenarios and modalities. Extensive experiments on various vision-language downstream tasks demonstrate the effectiveness of Knowledge-CLIP compared with the original CLIP and competitive baselines.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 49 of its 82 references that are in the map. First posted on arXiv 17 Oct 2022.

Builds on

49 of 82 references.

References · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  8. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  9. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
37 more
  1. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  2. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  3. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  4. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  5. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  6. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  7. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  8. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  9. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  10. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  11. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  12. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  13. 2017Neural Motifs: Scene Graph Parsing with Global Contextcited by 413
  14. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  15. 2018Graph R-CNN for Scene Graph Generationcited by 296
  16. 2022Vision Transformer with Deformable Attentioncited by 293
  17. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  18. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 225
  19. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  20. 20203D Object Detection with Pointformercited by 194
  21. 2017Variational Reasoning for Question Answering with Knowledge Graphcited by 170
  22. 2020A Survey on Knowledge Graph-Based Recommender Systemscited by 170
  23. 2018RippleNet: Propagating User Preferences on the Knowledge Graph for Recommender Systemscited by 136
  24. 2021Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Taskscited by 124
  25. 2018Rethinking Knowledge Graph Propagation for Zero-Shot Learningcited by 87
  26. 2019Fusion of Detected Objects in Text for Visual Question Answeringcited by 74
  27. 2021On the Integration of Self-Attention and Convolutioncited by 74
  28. 2020JAKET: Joint Pre-training of Knowledge Graph and Language Understandingcited by 73
  29. 2021Kaleido-BERT: Vision-Language Pre-training on Fashion Domaincited by 67
  30. 2017Open-World Knowledge Graph Completioncited by 64
  31. 2018Deep Reasoning with Knowledge Graph for Social Relationship Understandingcited by 44
  32. 2021Boosting Entity-aware Image Captioning with Multi-modal Knowledge Graphcited by 31
  33. 2020VisualSem: A High-quality Knowledge Graph for Vision and Languagecited by 30
  34. 2018Variational Knowledge Graph Reasoningcited by 27
  35. 2020Bidirectional Graph Reasoning Network for Panoptic Segmentationcited by 26
  36. 2021Knowledge Perceived Multi-modal Pretraining in E-commercecited by 18
  37. 2020End-to-End Entity Classification on Multimodal Knowledge Graphscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 12; 2024: 6; 2025: 9; 2026: 6.

34 papers.

Sort:

By area:

  1. 2023FLatten Transformer: Vision Transformer using Focused Linear Attentioncited by 170
  2. 2022A Survey of Knowledge Graph Reasoning on Graph Types: Static, Dynamic, and Multimodalcited by 106
  3. 2023Agent Attention: On the Integration of Softmax and Linear Attentioncited by 90
  4. 2024Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Surveycited by 50
  5. 2023Slide-Transformer: Hierarchical Vision Transformer with Local Self-Attentioncited by 38
  6. 2023Pre-trained Trojan Attacks for Visual Recognitioncited by 38
  7. 2024Heterogeneous Contrastive Learning for Foundation Models and Beyondcited by 27
  8. 2023Does Progress On Object Recognition Benchmarks Improve Real-World Generalization?cited by 13
  9. 2024Grounding Language Models for Visual Entity Recognitioncited by 12
  10. 2023Towards Medical Artificial General Intelligence via Knowledge-Enhanced Multimodal Pretrainingcited by 11
  11. 2023Efficient Token-Guided Image-Text Retrieval with Consistent Multimodal Contrastive Trainingcited by 10
  12. 2023Structural Information Guided Multimodal Pre-training for Vehicle-centric Perceptioncited by 7
22 more
  1. 2024Noise-powered Multi-modal Knowledge Graph Representation Frameworkcited by 7
  2. 2025EvdCLIP: Improving Vision-Language Retrieval with Entity Visual Descriptions from Large Language Modelscited by 6
  3. 2023Knowledge Graphs Queryingcited by 5
  4. 2023Language Semantic Graph Guided Data-Efficient Learningcited by 4
  5. 2025AnchorFormer: Differentiable Anchor Attention for Efficient Vision Transformercited by 3
  6. 2025Schema Generation for Large Knowledge Graphs Using Large Language Modelscited by 3
  7. 2023River of No Return: Graph Percolation Embeddings for Efficient Knowledge Graph Reasoningcited by 2
  8. 2023Contextualized Structural Self-supervised Learning for Ontology Matchingcited by 2
  9. 2025Does CLIP perceive art the same way we do?cited by 2
  10. 2025MEDMKG: Benchmarking Medical Knowledge Exploitation with Multimodal Knowledge Graphcited by 2
  11. 2026VL-KGE: Vision-Language Models Meet Knowledge Graph Embeddingscited by 2
  12. 2024Learning Generalized Medical Image Representations through Image-Graph Contrastive Pretrainingcited by 1
  13. 2025SchoenbAt: Rethinking Attention with Polynomial basiscited by 1
  14. 2026HyPER-GAN: Hybrid Patch-Based Image-to-Image Translation for Real-Time Photorealism Enhancement in Game Enginescited by 1
  15. 2026Art Beyond Semantics: Sheaf-Informed Contrastive Learning for Multi-Relational Representationscited by 1
  16. 2024CUE-M: Contextual Understanding and Enhanced Search with Multimodal Large Language Modelcited by 0
  17. 2025Tri-FusionNet: Enhancing Image Description Generation with Transformer-based Fusion Network and Dual Attention Mechanismcited by 0
  18. 2025Advanced Chest X-Ray Analysis via Transformer-Based Image Descriptors and Cross-Model Attention Mechanismcited by 0
  19. 2025Vision Transformers: the threat of realistic adversarial patchescited by 0
  20. 2026A Survey on Semantic Communication for Vision: Categories, Frameworks, Enabling Techniques, and Applicationscited by 0
  21. 2026Thermal-Det: Language-Guided Cross-Modal Distillation for Open-Vocabulary Thermal Object Detectioncited by 0
  22. 2026TAME: Temporal-Aware Mixture-of-Experts for Text-Video Retrievalcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024CLIP-CID: Efficient CLIP Distillation via Cluster-Instance Discrimination
  2. 2022CLIP-TD: CLIP Targeted Distillation for Vision-Language Tasks
  3. 2026Sparse CLIP: Co-Optimizing Interpretability and Performance in Contrastive Learning
  4. 2023Structure-CLIP: Towards Scene Graph Knowledge to Enhance Multi-modal Structured Representations
  5. 2023A Closer Look at the Explainability of Contrastive Language-Image Pre-training
  6. 2025Post-pre-training for Modality Alignment in Vision-Language Foundation Models
  7. 2023CLIP meets Model Zoo Experts: Pseudo-Supervision for Visual Enhancement
  8. 2025Dynamic Knowledge Integration for Enhanced Vision-Language Reasoning