mlsanityData to 7 Oct 2026

UKnow: A Unified Knowledge Protocol with Multimodal Knowledge Graph Datasets for Reasoning and Vision-Language Pre-Training

arXiv 2302.06891 ↗First posted 14 Feb 2023cs.CVcited by 14

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

This work presents a unified knowledge protocol, called UKnow, which facilitates knowledge-based studies from the perspective of data. Particularly focusing on visual and linguistic modalities, we categorize data knowledge into five unit types, namely, in-image, in-text, cross-image, cross-text, and image-text, and set up an efficient pipeline to help construct the multimodal knowledge graph from any data collection. Thanks to the logical information naturally contained in knowledge graph, organizing datasets under UKnow format opens up more possibilities of data usage compared to the commonly used image-text pairs. Following UKnow protocol, we collect, from public international news, a large-scale multimodal knowledge graph dataset that consists of 1,388,568 nodes (with 571,791 vision-related ones) and 3,673,817 triplets. The dataset is also annotated with rich event tags, including 11 coarse labels and 9,185 fine labels. Experiments on 4 benchmarks demonstrate the potential of UKnow in supporting common-sense reasoning and boosting vision-language pre-training with a single dataset, benefiting from its unified form of knowledge organization. See Appendix to download the dataset.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 66 of its 102 references that are in the map. First posted on arXiv 14 Feb 2023.

Builds on

66 of 102 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Vision-Language Pre-Training with Triple Contrastive Learning21 Feb
  2. 2021Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training25 Jun
  3. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning7 Apr

Other references · 63

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  6. 2017Inductive Representation Learning on Large Graphscited by 6,284
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  10. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
38 more
  1. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  2. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  3. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  4. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  5. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  6. 2021Grounded Language-Image Pre-trainingcited by 1,147
  7. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  8. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  9. 2016Complex Embeddings for Simple Link Predictioncited by 897
  10. 2019RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Spacecited by 893
  11. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  12. 2019Self-supervised Visual Feature Learning with Deep Neural Networks: A Surveycited by 656
  13. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  14. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 419
  15. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  16. 2021QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answeringcited by 386
  17. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  18. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  19. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  20. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  21. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  22. 2017DeepPath: A Reinforcement Learning Method for Knowledge Graph Reasoningcited by 265
  23. 2020Self-supervised Co-training for Video Representation Learningcited by 232
  24. 2020Query2box: Reasoning over Knowledge Graphs in Vector Space using Box Embeddingscited by 169
  25. 2020Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgettingcited by 169
  26. 2021Zero-Shot Out-of-Distribution Detection Based on the Pre-trained Model CLIPcited by 151
  27. 2018End-to-end Structure-Aware Convolutional Networks for Knowledge Base Completioncited by 136
  28. 2020Beta Embeddings for Multi-Hop Logical Reasoning in Knowledge Graphscited by 128
  29. 2020KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQAcited by 124
  30. 2018Embedding Logical Queries on Knowledge Graphscited by 120
  31. 2020A Benchmarking Study of Embedding-based Entity Alignment for Knowledge Graphscited by 119
  32. 2022CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailmentcited by 113
  33. 2021Aligning Pretraining for Detection via Object-Level Contrastive Learningcited by 110
  34. 2019MMKG: Multi-Modal Knowledge Graphscited by 104
  35. 2021Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-supervised Action Recognitioncited by 89
  36. 2022CLIP-Event: Connecting Text and Images with Event Structurescited by 86
  37. 2022Multi-Modal Knowledge Graph Construction and Application: A Surveycited by 74
  38. 2022Fine-grained Image Captioning with CLIP Rewardcited by 69

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 7.

14 papers.

Sort:

By area:

  1. 2024Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Surveycited by 50
  2. 2026PhysRVG: Physics-Aware Unified Reinforcement Learning for Video Generative Modelscited by 15
  3. 2025Ming-Lite-Uni: Advancements in Unified Architecture for Natural Multimodal Interactioncited by 13
  4. 2025SynMotion: Semantic-Visual Adaptation for Motion Customized Video Generationcited by 7
  5. 2025A Survey of Link Prediction in Temporal Networkscited by 6
  6. 2025Rethinking Regularization Methods for Knowledge Graph Completioncited by 6
  7. 2026RoboRouter: Training-Free Policy Routing for Robotic Manipulationcited by 6
  8. 2025VAT-KG: Knowledge-Intensive Multimodal Knowledge Graph Dataset for Retrieval-Augmented Generationcited by 2
  9. 2026VL-KGE: Vision-Language Models Meet Knowledge Graph Embeddingscited by 2
  10. 2026From Corpora to Co-Evolving Capabilities: Capability-Centric Data Design for Generalist Image Generationcited by 1
  11. 2025VisKnow: Constructing Visual Knowledge Base for Object Understandingcited by 0
  12. 2026Life-Bench: A Benchmark and Knowledge Graph Framework for Multimodal Personalization Beyond Concept Recognitioncited by 0
2 more
  1. 2026Event-Centric Human Value Understanding in News-Domain Texts: An Actor-Conditioned, Multi-Granularity Benchmarkcited by 0
  2. 2026VISPATH: Visual-Intent-Guided Path Reasoning for Multimodal Knowledge Graph Question Answeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025VKnowU: Evaluating Visual Knowledge Understanding in Multimodal LLMs
  2. 2026SmoGVLM: A Small, Graph-enhanced Vision-Language Model
  3. 2025Aligning Vision to Language: Annotation-Free Multimodal Knowledge Graph Construction for Enhanced LLMs Reasoning
  4. 2025ImageNet-Think-250K: A Large-Scale Synthetic Dataset for Multimodal Reasoning for Vision Language Models
  5. 2025Dynamic Knowledge Integration for Enhanced Vision-Language Reasoning
  6. 2026GraphVerse: A Comprehensive Visual Graph Reasoning Benchmark for Multimodal Large Language Models
  7. 2026ViSR-KGC: Visual Subgraph Reasoning with Vision-Language Models for Multimodal Knowledge Graph Completion
  8. 2025Simple o3: Towards Interleaved Vision-Language Reasoning