mlsanityData to 14 Oct 2026

A Unified Continuous Learning Framework for Multi-modal Knowledge Discovery and Pre-training

arXiv 2206.05555 ↗First posted 11 Jun 2022cs.CL cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

Multi-modal pre-training and knowledge discovery are two important research topics in multi-modal machine learning. Nevertheless, none of existing works make attempts to link knowledge discovery with knowledge guided multi-modal pre-training. In this paper, we propose to unify them into a continuous learning framework for mutual improvement. Taking the open-domain uni-modal datasets of images and texts as input, we maintain a knowledge graph as the foundation to support these two tasks. For knowledge discovery, a pre-trained model is used to identify cross-modal links on the graph. For model pre-training, the knowledge graph is used as the external knowledge to guide the model updating. These two steps are iteratively performed in our framework for continuous learning. The experimental results on MS-COCO and Flickr30K with respect to both knowledge discovery and the pre-trained model validate the effectiveness of our framework.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 39 of its 50 references that are in the map. First posted on arXiv 11 Jun 2022.

Builds on

39 of 50 references.

References · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  7. 2015Fast R-CNNcited by 2,933
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  9. 2015VQA: Visual Question Answeringcited by 2,506
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  11. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  12. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
27 more
  1. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  2. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  3. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  4. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  5. 2016Modeling Context in Referring Expressionscited by 1,063
  6. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  7. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 528
  8. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  9. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  10. 2019KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representationcited by 356
  11. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 319
  12. 2019K-BERT: Enabling Language Representation with Knowledge Graphcited by 304
  13. 2019ERNIE: Enhanced Representation through Knowledge Integrationcited by 302
  14. 2021In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learningcited by 301
  15. 2019Label Propagation for Deep Semi-supervised Learningcited by 289
  16. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  17. 2019MMKG: Multi-Modal Knowledge Graphscited by 105
  18. 2020Contrastive Learning for Weakly Supervised Phrase Groundingcited by 102
  19. 2020Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillationcited by 51
  20. 2022Hybrid Transformer with Multi-level Fusion for Multimodal Knowledge Graph Completioncited by 44
  21. 2018Embedding Multimodal Relational Data for Knowledge Base Completioncited by 34
  22. 2015Evaluating Link Prediction Methodscited by 31
  23. 2021ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integrationcited by 31
  24. 2020VisualSem: A High-quality Knowledge Graph for Vision and Languagecited by 30
  25. 2019An Open-World Extension to Knowledge Graph Completion Modelscited by 16
  26. 2021Constructing Phrase-level Semantic Labels to Form Multi-Grained Supervision for Image-Text Retrievalcited by 4
  27. 2021Negative Sample is Negative in Its Own Way: Tailoring Negative Sentences for Image-Text Retrievalcited by 2

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 3.

5 papers.

  1. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  2. 2023Continual Vision-Language Representation Learning with Off-Diagonal Informationcited by 40
  3. 2024The Inter-Intra Modal Measure: A Predictive Lens on Fine-Tuning Outcomes in Vision-Language Modelscited by 1
  4. 2024Quantified Task Misalignment to Inform PEFT: An Exploration of Domain Generalization and Catastrophic Forgetting in CLIPcited by 0
  5. 2024E-CGL: An Efficient Continual Graph Learnercited by 0