Reminding Multimodal Large Language Models of Object-aware Knowledge with Retrieved Tags
On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning
Abstract (arXiv)
Despite recent advances in the general visual instruction-following ability of Multimodal Large Language Models (MLLMs), they still struggle with critical problems when required to provide a precise and detailed response to a visual instruction: (1) failure to identify novel objects or entities, (2) mention of non-existent objects, and (3) neglect of object's attributed details. Intuitive solutions include improving the size and quality of data or using larger foundation models. They show effectiveness in mitigating these issues, but at an expensive cost of collecting a vast amount of new data and introducing a significantly larger model. Standing at the intersection of these approaches, we examine the three object-oriented problems from the perspective of the image-to-text mapping process by the multimodal connector. In this paper, we first identify the limitations of multimodal connectors stemming from insufficient training data. Driven by this, we propose to enhance the mapping with retrieval-augmented tag tokens, which contain rich object-aware information such as object names and attributes. With our Tag-grounded visual instruction tuning with retrieval Augmentation (TUNA), we outperform baselines that share the same language model and training data on 12 benchmarks. Furthermore, we show the zero-shot capability of TUNA when provided with specific datastores.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 55 references that are in the map. First posted on arXiv 16 Jun 2024.
Builds on
36 of 55 references.
Closest ancestors · 6
Same direction, newest first
- 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
- 2023Improved Baselines with Visual Instruction Tuning5 Oct
- 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
- 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
- 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
- 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May
Other references · 30
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2014Microsoft COCO: Common Objects in Contextcited by 17,498
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
- 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
- 2017Billion-scale similarity search with GPUscited by 2,477
- 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
- 2019Towards VQA Models That Can Readcited by 1,696
- 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
- 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
- 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
- 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
18 more
- 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
- 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
- 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
- 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
- 2016Image Captioning with Semantic Attentioncited by 260
- 2017Automatic Spatially-aware Fashion Concept Discoverycited by 132
- 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memorycited by 122
- 2023Tag2Text: Guiding Vision-Language Model via Image Taggingcited by 84
- 2022SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentationcited by 82
- 2018Fashion-Gen: The Generative Fashion Dataset and Challengecited by 65
- 2018Learning Type-Aware Embeddings for Fashion Compatibilitycited by 61
- 2023Better Generative Replay for Continual Federated Learningcited by 53
- 2023FACTUAL: A Benchmark for Faithful and Consistent Textual Scene Graph Parsingcited by 50
- 2023Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioningcited by 41
- 2023EVCap: Retrieval-Augmented Image Captioning with External Visual-Name Memory for Open-World Comprehensioncited by 32
- 2015What value do explicit high level concepts have in vision to language problems?cited by 31
- 2022IDEA: Increasing Text Diversity via Online Multi-Label Recognition for Vision-Language Pre-trainingcited by 14
- 2020Less is More: Data-Efficient Complex Question Answering over Knowledge Basescited by 9
Built on by
Citing papers per year: 2024: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2024List Items One by One: A New Data Source and Learning Paradigm for Multimodal LLMs
- 2025Region-Level Context-Aware Multimodal Understanding
- 2024MMEvol: Empowering Multimodal Large Language Models with Evol-Instruct
- 2025Visual Representation Alignment for Multimodal Large Language Models
- 2024MLAN: Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Models
- 2024Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Models
- 2023Text as Images: Can Multimodal Large Language Models Follow Printed Instructions in Pixels?
- 2024Rethinking Visual Prompting for Multimodal Large Language Models with External Knowledge