mlsanityData to 7 Oct 2026

ImageRef-VL: Enabling Contextual Image Referencing in Vision-Language Models

arXiv 2501.12418 ↗First posted 20 Jan 2025cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

also in 1 more
Abstract (arXiv)

Vision-Language Models (VLMs) have demonstrated remarkable capabilities in understanding multimodal inputs and have been widely integrated into Retrieval-Augmented Generation (RAG) based conversational systems. While current VLM-powered chatbots can provide textual source references in their responses, they exhibit significant limitations in referencing contextually relevant images during conversations. In this paper, we introduce Contextual Image Reference -- the ability to appropriately reference relevant images from retrieval documents based on conversation context -- and systematically investigate VLMs' capability in this aspect. We conduct the first evaluation for contextual image referencing, comprising a dedicated testing dataset and evaluation metrics. Furthermore, we propose ImageRef-VL, a method that significantly enhances open-source VLMs' image referencing capabilities through instruction fine-tuning on a large-scale, manually curated multimodal conversation dataset. Experimental results demonstrate that ImageRef-VL not only outperforms proprietary models but also achieves an 88% performance improvement over state-of-the-art open-source VLMs in contextual image referencing tasks. Our code is available at https://github.com/bytedance/ImageRef-VL.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 36 references that are in the map. First posted on arXiv 20 Jan 2025.

Builds on

24 of 36 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  8. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  10. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  11. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  12. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
12 more
  1. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  2. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  3. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 247
  4. 2024LongCite: Enabling LLMs to Generate Fine-grained Citations in Long-context QAcited by 78
  5. 2023Effective Large Language Model Adaptation for Improved Grounding and Citation Generationcited by 44
  6. 2024Training Language Models to Generate Text with Citations via Fine-grained Rewardscited by 40
  7. 2024Learning to Plan and Generate Text with Citationscited by 28
  8. 2024Improving Attributed Text Generation of Large Language Models via Preference Learningcited by 23
  9. 2023Towards Verifiable Text Generation with Symbolic Referencescited by 13
  10. 2024Verifiable by Design: Aligning Language Models to Quote from Pre-Training Datacited by 13
  11. 2023Towards Reliable and Fluent Large Language Models: Incorporating Feedback Learning Loops in QA Systemscited by 10
  12. 2024Citekit: A Modular Toolkit for Large Language Model Citation Generationcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Resolving References in Visually-Grounded Dialogue via Text Generation
  2. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  3. 2024SURf: Teaching Large Vision-Language Models to Selectively Utilize Retrieved Information
  4. 2024RoRA-VLM: Robust Retrieval-Augmented Vision Language Models
  5. 2025VisCon-100K: Leveraging Contextual Web Data for Fine-tuning Vision Language Models
  6. 2023Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning
  7. 2025Taming a Retrieval Framework to Read Images in Humanlike Manner for Augmenting Generation of MLLMs
  8. 2026WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering