# Multimodal Hypothetical Summary for Retrieval-based Multi-image Question Answering

arXiv: https://arxiv.org/abs/2412.14880 · HTML: https://mlsanity.com/paper/2412.14880/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 1 of its 40 references that are in the map.
- First posted on arXiv 19 Dec 2024.

## Builds on (most cited first, 1 of 1)

- [ARAGOG: Advanced RAG Output Grading](https://mlsanity.com/paper/2404.01037/) (2024)

## Built on by (most cited first, 1 of 1)

- [Exploring and Evaluating Multimodal Knowledge Reasoning Consistency of Multimodal Large Language Models](https://mlsanity.com/paper/2503.04801/) (2025)

## Abstract (arXiv)

Retrieval-based multi-image question answering (QA) task involves retrieving multiple question-related images and synthesizing these images to generate an answer. Conventional "retrieve-then-answer" pipelines often suffer from cascading errors because the training objective of QA fails to optimize the retrieval stage. To address this issue, we propose a novel method to effectively introduce and reference retrieved information into the QA. Given the image set to be retrieved, we employ a multimodal large language model (visual perspective) and a large language model (textual perspective) to obtain multimodal hypothetical summary in question-form and description-form. By combining visual and textual perspectives, MHyS captures image content more specifically and replaces real images in retrieval, which eliminates the modality gap by transforming into text-to-text retrieval and helps improve retrieval. To more advantageously introduce retrieval with QA, we employ contrastive learning to align queries (questions) with MHyS. Moreover, we propose a coarse-to-fine strategy for calculating both sentence-level and word-level similarity scores, to further enhance retrieval and filter out irrelevant details. Our approach achieves a 3.7% absolute improvement over state-of-the-art methods on RETVQA and a 14.5% improvement over CLIP. Comprehensive experiments and detailed ablation studies demonstrate the superiority of our method.