# Explainable Search and Discovery of Visual Cultural Heritage Collections with Multimodal Large Language Models

arXiv: https://arxiv.org/abs/2411.04663 · HTML: https://mlsanity.com/paper/2411.04663/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 17 of its 51 references that are in the map.
- First posted on arXiv 7 Nov 2024.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 17 of 17)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs](https://mlsanity.com/paper/2401.06209/) (2024)
- [Ferret: Refer and Ground Anything Anywhere at Any Granularity](https://mlsanity.com/paper/2310.07704/) (2023)
- [GPT4Vis: What Can GPT-4 Do for Zero-shot Visual Recognition?](https://mlsanity.com/paper/2311.15732/) (2023)
- [Text Clustering with Large Language Model Embeddings](https://mlsanity.com/paper/2403.15112/) (2024)
- [No Token Left Behind: Explainability-Aided Image Classification and Generation](https://mlsanity.com/paper/2204.04908/) (2022)
- [A Friendly Face: Do Text-to-Image Systems Rely on Stereotypes when the Input is Under-Specified?](https://mlsanity.com/paper/2302.07159/) (2023)
- [ChatGPT's One-year Anniversary: Are Open-Source Large Language Models Catching up?](https://mlsanity.com/paper/2311.16989/) (2023)
- [Embedding Deep Networks into Visual Explanations](https://mlsanity.com/paper/1709.05360/) (2017)
- [LICO: Explainable Models with Language-Image Consistency](https://mlsanity.com/paper/2310.09821/) (2023)
- [Visualizing Digital Collections](https://mlsanity.com/paper/1402.2695/) (2014)
- [Semiotically-grounded distant viewing of diagrams: insights from two multimodal corpora](https://mlsanity.com/paper/2103.04692/) (2021)

## Built on by (most cited first, 1 of 1)

- [On the Cultural Anachronism and Temporal Reasoning in Vision Language Models](https://mlsanity.com/paper/2605.15071/) (2026)

## Abstract (arXiv)

Many cultural institutions have made large digitized visual collections available online, often under permissible re-use licences. Creating interfaces for exploring and searching these collections is difficult, particularly in the absence of granular metadata. In this paper, we introduce a method for using state-of-the-art multimodal large language models (LLMs) to enable an open-ended, explainable search and discovery interface for visual collections. We show how our approach can create novel clustering and recommendation systems that avoid common pitfalls of methods based directly on visual embeddings. Of particular interest is the ability to offer concrete textual explanations of each recommendation without the need to preselect the features of interest. Together, these features can create a digital interface that is more open-ended and flexible while also being better suited to addressing privacy and ethical concerns. Through a case study using a collection of documentary photographs, we provide several metrics showing the efficacy and possibilities of our approach.