mlsanityData to 14 Oct 2026

LaKo: Knowledge-driven Visual Question Answering via Late Knowledge-to-Text Injection

arXiv 2207.12888 ↗First posted 26 Jul 2022cs.CV cs.AIcited by 18

Not on the map. Why?

Abstract (arXiv)

Visual question answering (VQA) often requires an understanding of visual concepts and language semantics, which relies on external knowledge. Most existing methods exploit pre-trained language models or/and unstructured text, but the knowledge in these resources are often incomplete and noisy. Some other methods prefer to use knowledge graphs (KGs) which often have intensive structured knowledge, but the research is still quite preliminary. In this paper, we propose LaKo, a knowledge-driven VQA method via Late Knowledge-to-text Injection. To effectively incorporate an external KG, we transfer triples into textual format and propose a late injection mechanism for knowledge fusion. Finally we address VQA as a text generation task with an effective encoder-decoder paradigm, which achieves state-of-the-art results on OKVQA dataset.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 54 of its 70 references that are in the map. First posted on arXiv 26 Jul 2022.

Builds on

54 of 70 references.

References · 54

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  9. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  10. 2015VQA: Visual Question Answeringcited by 2,506
  11. 2017Billion-scale similarity search with GPUscited by 2,497
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
38 more
  1. 2019Language Models as Knowledge Bases?cited by 1,936
  2. 2017Modeling Relational Data with Graph Convolutional Networkscited by 1,694
  3. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  4. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  5. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  6. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  7. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  8. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  9. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  10. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,067
  11. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  12. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 719
  13. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  14. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  15. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 301
  16. 2018Bilinear Attention Networkscited by 280
  17. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  18. 2016COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Imagescited by 236
  19. 2020Distilling Knowledge from Reader to Retriever for Question Answeringcited by 226
  20. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  21. 2016FVQA: Fact-based Visual Question Answeringcited by 181
  22. 2020How Context Affects Language Models' Factual Predictionscited by 181
  23. 2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 126
  24. 2020KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQAcited by 125
  25. 2020Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queriescited by 107
  26. 2021Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Basescited by 107
  27. 2021Multi-Modal Answer Validation for Knowledge-Based VQAcited by 79
  28. 2022K-LITE: Learning Transferable Visual Models with External Knowledgecited by 79
  29. 2017Tips and Tricks for Visual Question Answering: Learnings from the 2017 Challengecited by 76
  30. 2022REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answeringcited by 59
  31. 2020Mucko: Multi-Layer Cross-Modal Knowledge Reasoning for Fact-based Visual Question Answeringcited by 51
  32. 2021Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answeringcited by 50
  33. 2021OntoZSL: Ontology-enhanced Zero-shot Learningcited by 45
  34. 2021Knowledge-aware Zero-Shot Learning: Survey and Perspectivecited by 41
  35. 2021Zero-shot Visual Question Answering using Knowledge Graphcited by 37
  36. 2021Select, Substitute, Search: A New Benchmark for Knowledge-Augmented Visual Question Answeringcited by 33
  37. 2021Passage Retrieval for Outside-Knowledge Visual Question Answeringcited by 25
  38. 2021Image Captioning for Effective Use of Language Models in Knowledge-Based Visual Question Answeringcited by 25

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 10; 2024: 3; 2025: 2.

18 papers.

Sort:

By area:

  1. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 668
  2. 2023Large Language Models are Visual Reasoning Coordinatorscited by 58
  3. 2023Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answeringcited by 51
  4. 2023Making Large Language Models Perform Better in Knowledge Graph Completioncited by 48
  5. 2024PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrieverscited by 42
  6. 2023Structure-CLIP: Towards Scene Graph Knowledge to Enhance Multi-modal Structured Representationscited by 37
  7. 2022MEAformer: Multi-modal Entity Alignment Transformer for Meta Modality Hybridcited by 30
  8. 2023Knowledgeable Preference Alignment for LLMs in Domain-specific Question Answeringcited by 29
  9. 2023Rethinking Uncertainly Missing and Ambiguous Visual Modality in Multi-Modal Entity Alignmentcited by 21
  10. 2022Tele-Knowledge Pre-training for Fault Analysiscited by 18
  11. 2023A Symmetric Dual Encoding Dense Retrieval Framework for Knowledge-Intensive Visual Question Answeringcited by 14
  12. 2023Pre-Training Multi-Modal Dense Retrievers for Outside-Knowledge Visual Question Answeringcited by 10
6 more
  1. 2025NAVIG: Natural Language-guided Analysis with Vision Language Models for Image Geo-localizationcited by 8
  2. 2024DRAK: Unlocking Molecular Insights with Domain-Specific Retrieval-Augmented Knowledge in LLMscited by 3
  3. 2022Target-oriented Sentiment Classification with Sequential Cross-modal Semantic Graphcited by 2
  4. 2023The Contribution of Knowledge in Visiolinguistic Learning: A Survey on Tasks and Challengescited by 2
  5. 2024Is Large Language Model Good at Triple Set Prediction? An Empirical Studycited by 1
  6. 2025Augmented Vision-Language Models: A Systematic Reviewcited by 1