mlsanityData to 7 Oct 2026

Leveraging Chat-Based Large Vision Language Models for Multimodal Out-Of-Context Detection

arXiv 2403.08776 ↗First posted 22 Jan 2024cs.CV cs.AIcited by 4

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Out-of-context (OOC) detection is a challenging task involving identifying images and texts that are irrelevant to the context in which they are presented. Large vision-language models (LVLMs) are effective at various tasks, including image classification and text generation. However, the extent of their proficiency in multimodal OOC detection tasks is unclear. In this paper, we investigate the ability of LVLMs to detect multimodal OOC and show that these models cannot achieve high accuracy on OOC detection tasks without fine-tuning. However, we demonstrate that fine-tuning LVLMs on multimodal OOC datasets can further improve their OOC detection accuracy. To evaluate the performance of LVLMs on OOC detection tasks, we fine-tune MiniGPT-4 on the NewsCLIPpings dataset, a large dataset of multimodal OOC. Our results show that fine-tuning MiniGPT-4 on the NewsCLIPpings dataset significantly improves the OOC detection accuracy in this dataset. This suggests that fine-tuning can significantly improve the performance of LVLMs on OOC detection tasks.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 20 of its 31 references that are in the map. First posted on arXiv 22 Jan 2024.

Builds on

20 of 31 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 19

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  9. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  10. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  11. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  12. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
7 more
  1. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  2. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 146
  3. 2020Visual News: Benchmark and Challenges in News Image Captioningcited by 121
  4. 2021NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Mediacited by 106
  5. 2021Open-Domain, Content-based, Multi-modal Fact-checking of Out-of-Context Images via Online Resourcescited by 71
  6. 2023Interpretable Detection of Out-of-Context Misinformation with Neural-Symbolic-Enhanced Large Multimodal Modelcited by 8
  7. 2023Detecting Out-of-Context Image-Caption Pairs in News: A Counter-Intuitive Methodcited by 2

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 3.

4 papers.

  1. 2025XFacta: Contemporary, Real-World Dataset and Evaluation for Multimodal Misinformation Detection with Multimodal LLMscited by 6
  2. 2024Learning Domain-Invariant Features for Out-of-Context News Detectioncited by 3
  3. 2025Multi-MLLM Knowledge Distillation for Out-of-Context News Detectioncited by 2
  4. 2025Zero-Shot Warning Generation for Misinformative Multimodal Contentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Models
  2. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Review
  3. 2025Beyond CNNs: Efficient Fine-Tuning of Multi-Modal LLMs for Object Detection on Low-Data Regimes
  4. 2025E2LVLM:Evidence-Enhanced Large Vision-Language Model for Multimodal Out-of-Context Misinformation Detection
  5. 2024SNIFFER: Multimodal Large Language Model for Explainable Out-of-Context Misinformation Detection
  6. 2023Contextual Object Detection with Multimodal Large Language Models
  7. 2026FindIt: A Format-Informed Visual Detection Benchmark for Generalist Multimodal LLMs
  8. 2025Harnessing Large Language and Vision-Language Models for Robust Out-of-Distribution Detection