mlsanityData to 14 Oct 2026

LLaVA-RE: Binary Image-Text Relevancy Evaluation with Multimodal Large Language Model

arXiv 2508.05602 ↗First posted 7 Aug 2025cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal generative AI usually involves generating image or text responses given inputs in another modality. The evaluation of image-text relevancy is essential for measuring response quality or ranking candidate responses. In particular, binary relevancy evaluation, i.e., ``Relevant'' vs. ``Not Relevant'', is a fundamental problem. However, this is a challenging task considering that texts have diverse formats and the definition of relevancy varies in different scenarios. We find that Multimodal Large Language Models (MLLMs) are an ideal choice to build such evaluators, as they can flexibly handle complex text formats and take in additional task information. In this paper, we present LLaVA-RE, a first attempt for binary image-text relevancy evaluation with MLLM. It follows the LLaVA architecture and adopts detailed task instructions and multimodal in-context samples. In addition, we propose a novel binary relevancy data set that covers various tasks. Experimental results validate the effectiveness of our framework.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 15 of its 29 references that are in the map. First posted on arXiv 7 Aug 2025.

Builds on

15 of 29 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  2. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  3. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning14 Sep

Other references · 12

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2019Towards VQA Models That Can Readcited by 1,718
  6. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  7. 2018Stacked Cross Attention for Image-Text Matchingcited by 416
  8. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 287
  9. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  10. 2017Learning Two-Branch Neural Networks for Image-Text Matching Taskscited by 119
  11. 2017An Analysis of Visual Question Answering Algorithmscited by 90
  12. 2018RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipescited by 74

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Multimodal Language Models as Text-to-Image Model Evaluators
  2. 2026Bi-Level Prompt Optimization for Multimodal LLM-as-a-Judge
  3. 2025Multimodal LLMs as Customized Reward Models for Text-to-Image Generation
  4. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models
  5. 2023MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria
  6. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs
  7. 2025MINOS: A Multimodal Evaluation Model for Bidirectional Generation Between Image and Text
  8. 2024EVALALIGN: Supervised Fine-Tuning Multimodal LLMs with Human-Aligned Data for Evaluating Text-to-Image Models