# LLaVA-RE: Binary Image-Text Relevancy Evaluation with Multimodal Large Language Model

arXiv: https://arxiv.org/abs/2508.05602 · HTML: https://mlsanity.com/paper/2508.05602/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 15 of its 29 references that are in the map.
- First posted on arXiv 7 Aug 2025.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 15 of 15)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [Stacked Cross Attention for Image-Text Matching](https://mlsanity.com/paper/1803.08024/) (2018)
- [WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning](https://mlsanity.com/paper/2103.01913/) (2021)
- [MIMIC-IT: Multi-Modal In-Context Instruction Tuning](https://mlsanity.com/paper/2306.05425/) (2023)
- [MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning](https://mlsanity.com/paper/2309.07915/) (2023)
- [Learning Two-Branch Neural Networks for Image-Text Matching Tasks](https://mlsanity.com/paper/1704.03470/) (2017)
- [An Analysis of Visual Question Answering Algorithms](https://mlsanity.com/paper/1703.09684/) (2017)
- [RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipes](https://mlsanity.com/paper/1809.00812/) (2018)

## Abstract (arXiv)

Multimodal generative AI usually involves generating image or text responses given inputs in another modality. The evaluation of image-text relevancy is essential for measuring response quality or ranking candidate responses. In particular, binary relevancy evaluation, i.e., ``Relevant'' vs. ``Not Relevant'', is a fundamental problem. However, this is a challenging task considering that texts have diverse formats and the definition of relevancy varies in different scenarios. We find that Multimodal Large Language Models (MLLMs) are an ideal choice to build such evaluators, as they can flexibly handle complex text formats and take in additional task information. In this paper, we present LLaVA-RE, a first attempt for binary image-text relevancy evaluation with MLLM. It follows the LLaVA architecture and adopts detailed task instructions and multimodal in-context samples. In addition, we propose a novel binary relevancy data set that covers various tasks. Experimental results validate the effectiveness of our framework.