The Contribution of Knowledge in Visiolinguistic Learning: A Survey on Tasks and Challenges
Not on the map. Why?
Abstract (arXiv)
Recent advancements in visiolinguistic (VL) learning have allowed the development of multiple models and techniques that offer several impressive implementations, able to currently resolve a variety of tasks that require the collaboration of vision and language. Current datasets used for VL pre-training only contain a limited amount of visual and linguistic knowledge, thus significantly limiting the generalization capabilities of many VL models. External knowledge sources such as knowledge graphs (KGs) and Large Language Models (LLMs) are able to cover such generalization gaps by filling in missing knowledge, resulting in the emergence of hybrid architectures. In the current survey, we analyze tasks that have benefited from such hybrid approaches. Moreover, we categorize existing knowledge sources and types, proceeding to discussion regarding the KG vs LLM dilemma and its potential impact to future hybrid approaches.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 117 of its 152 references that are in the map. First posted on arXiv 4 Mar 2023.
Builds on
117 of 152 references.
References · 117
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2014Microsoft COCO: Common Objects in Contextcited by 17,498
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
- 2021Zero-Shot Text-to-Image Generationcited by 4,276
- 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
- 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
- 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
38 more
- 2022Training Compute-Optimal Large Language Modelscited by 2,927
- 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
- 2022Emergent Abilities of Large Language Modelscited by 2,483
- 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
- 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
- 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
- 2019Language Models as Knowledge Bases?cited by 1,922
- 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
- 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
- 2021Improving language models by retrieving from trillions of tokenscited by 1,278
- 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
- 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
- 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
- 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
- 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
- 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
- 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
- 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
- 2019How Can We Know What Language Models Know?cited by 953
- 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
- 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
- 2021Florence: A New Foundation Model for Computer Visioncited by 802
- 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
- 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
- 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
- 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
- 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
- 2020A Survey on Knowledge Graphs: Representation, Acquisition and Applicationscited by 643
- 2022Towards Reasoning in Large Language Models: A Surveycited by 593
- 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
- 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
- 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
- 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
- 2020LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attentioncited by 326
- 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
- 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
- 2020COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphscited by 254
- 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 240
50 most cited of 117 All 117 (JSON)
Built on by
Citing papers per year: 2023: 2.
2 papers.