mlsanityData to 7 Oct 2026

KAT: A Knowledge Augmented Transformer for Vision-and-Language

arXiv 2112.08614 ↗First posted 16 Dec 2021cs.CLcited by 125

Not on the map. Why?

Cited by 22 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The primary focus of recent work with largescale transformers has been on optimizing the amount of information packed into the model's parameters. In this work, we ask a different question: Can multimodal transformers leverage explicit knowledge in their reasoning? Existing, primarily unimodal, methods have explored approaches under the paradigm of knowledge retrieval followed by answer prediction, but leave open questions about the quality and relevance of the retrieved knowledge used, and how the reasoning processes over implicit and explicit knowledge should be integrated. To address these challenges, we propose a novel model - Knowledge Augmented Transformer (KAT) - which achieves a strong state-of-the-art result (+6 points absolute) on the open-domain multimodal task of OK-VQA. Our approach integrates implicit and explicit knowledge in an end to end encoder-decoder architecture, while still jointly reasoning over both knowledge sources during answer generation. An additional benefit of explicit knowledge integration is seen in improved interpretability of model predictions in our analysis.

Key facts. Cited by 125 papers in the map (data through 7 Oct 2026). Builds on 29 of its 44 references that are in the map. First posted on arXiv 16 Dec 2021.

Builds on

29 of 44 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  10. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  11. 2015VQA: Visual Question Answeringcited by 2,489
  12. 2017Billion-scale similarity search with GPUscited by 2,477
17 more
  1. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  2. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  3. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  4. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  5. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  6. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  7. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  8. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  9. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  10. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  11. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  12. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
  13. 2016FVQA: Fact-based Visual Question Answeringcited by 181
  14. 2020KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQAcited by 124
  15. 2015Explicit Knowledge-based Reasoning for Visual Question Answeringcited by 85
  16. 2021Multi-Modal Answer Validation for Knowledge-Based VQAcited by 78
  17. 2021Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answeringcited by 50

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 17; 2023: 36; 2024: 40; 2025: 21; 2026: 11.

125 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  3. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  4. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  5. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 203
  6. 2022Rethinking with Retrieval: Faithful Large Language Model Inferencecited by 145
  7. 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memorycited by 122
  8. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  9. 2022From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Modelscited by 114
  10. 2023Encyclopedic VQA: Visual questions about detailed properties of fine-grained categoriescited by 105
  11. 2022PromptCap: Prompt-Guided Task-Aware Image Captioningcited by 104
  12. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 101
38 more
  1. 2023AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learncited by 97
  2. 2022Retrieval Augmented Visual Question Answering with Outside Knowledgecited by 93
  3. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
  4. 2023A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4cited by 81
  5. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 78
  6. 2023Prompting for Multimodal Hateful Meme Classificationcited by 68
  7. 2024Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agentcited by 65
  8. 2024EchoSight: Advancing Visual-Language Models with Wiki Knowledgecited by 61
  9. 2022REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answeringcited by 58
  10. 2023Large Language Models are Visual Reasoning Coordinatorscited by 58
  11. 2022LAVIS: A Library for Language-Vision Intelligencecited by 55
  12. 2024What Makes Multimodal In-Context Learning Work?cited by 54
  13. 2023Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answeringcited by 51
  14. 2023See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoningcited by 49
  15. 2024UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them Allcited by 49
  16. 2024PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrieverscited by 42
  17. 2022Neural Approaches to Conversational Information Retrievalcited by 41
  18. 2023Retrieval-Enhanced Contrastive Vision-Text Modelscited by 36
  19. 2024Situational Awareness Matters in 3D Vision Language Reasoningcited by 32
  20. 2023End-to-end Knowledge Retrieval with Multi-modal Queriescited by 23
  21. 2024Towards a Search Engine for Machines: Unified Ranking for Multiple Retrieval-Augmented Large Language Modelscited by 23
  22. 2023Prophet: Prompting Large Language Models with Complementary Answer Heuristics for Knowledge-based Visual Question Answeringcited by 21
  23. 2022METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signalscited by 20
  24. 2022VLC-BERT: Visual Question Answering with Contextualized Commonsense Knowledgecited by 20
  25. 2024Comparing Retrieval-Augmentation and Parameter-Efficient Fine-Tuning for Privacy-Preserving Personalization of Large Language Modelscited by 20
  26. 2023Improving Zero-shot Visual Question Answering via Large Language Models with Reasoning Question Promptscited by 19
  27. 2022LaKo: Knowledge-driven Visual Question Answering via Late Knowledge-to-Text Injectioncited by 18
  28. 2024SnapNTell: Enhancing Entity-Centric Visual Question Answering with Retrieval Augmented Multimodal LLMcited by 18
  29. 2025MAGIC-VQA: Multimodal And Grounded Inference with Commonsense Knowledge for Visual Question Answeringcited by 18
  30. 2023Generate then Select: Open-ended Visual Question Answering Guided by World Knowledgecited by 16
  31. 2024Modality-Aware Integration with Large Language Models for Knowledge-based Visual Question Answeringcited by 15
  32. 2024VLLMs Provide Better Context for Emotion Understanding Through Common Sense Reasoningcited by 15
  33. 2025MRAMG-Bench: A Comprehensive Benchmark for Advancing Multimodal Retrieval-Augmented Multimodal Generationcited by 15
  34. 2023A Symmetric Dual Encoding Dense Retrieval Framework for Knowledge-Intensive Visual Question Answeringcited by 14
  35. 2023A Simple Baseline for Knowledge-Based Visual Question Answeringcited by 14
  36. 2024Learning to Rank for Multiple Retrieval-Augmented Models through Iterative Utility Maximizationcited by 14
  37. 2024Accelerating Retrieval-Augmented Generationcited by 14
  38. 2024Cross-modal Retrieval for Knowledge-based Visual Question Answeringcited by 13

50 most cited of 125 All 125 (JSON)