# ILIAS: Instance-Level Image retrieval At Scale

arXiv: https://arxiv.org/abs/2502.11748 · HTML: https://mlsanity.com/paper/2502.11748/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 65 of its 90 references that are in the map.
- First posted on arXiv 17 Feb 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Document understanding and multimodal RAG](https://mlsanity.com/mllm/document-understanding-and-multimodal-rag/)

## Builds on (most cited first, 30 of 65)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [EVA-CLIP: Improved Training Techniques for CLIP at Scale](https://mlsanity.com/paper/2303.15389/) (2023)
- [EVA: Exploring the Limits of Masked Visual Representation Learning at Scale](https://mlsanity.com/paper/2211.07636/) (2022)
- [How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers](https://mlsanity.com/paper/2106.10270/) (2021)
- [Fine-tuning CNN Image Retrieval with No Human Annotation](https://mlsanity.com/paper/1711.02512/) (2017)
- [Deep Metric Learning via Lifted Structured Feature Embedding](https://mlsanity.com/paper/1511.06452/) (2015)
- [ResNet strikes back: An improved training procedure in timm](https://mlsanity.com/paper/2110.00476/) (2021)
- [CNN Features off-the-shelf: an Astounding Baseline for Recognition](https://mlsanity.com/paper/1403.6382/) (2014)
- [Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrieval](https://mlsanity.com/paper/2004.01804/) (2020)

## Built on by (most cited first, 30 of 30)

- [Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting](https://mlsanity.com/paper/2508.04227/) (2025)
- [Instance-Level Composed Image Retrieval](https://mlsanity.com/paper/2510.25387/) (2025)
- [ELIP: Enhanced Visual-Language Foundation Models for Image Retrieval](https://mlsanity.com/paper/2502.15682/) (2025)
- [REGLUE Your Latents with Global and Local Semantics for Entangled Diffusion](https://mlsanity.com/paper/2512.16636/) (2025)
- [Same or Not? Enhancing Visual Perception in Vision-Language Models](https://mlsanity.com/paper/2512.23592/) (2025)
- [Processing and acquisition traces in visual encoders: What does CLIP know about your camera?](https://mlsanity.com/paper/2508.10637/) (2025)
- [IsoCLIP: Decomposing CLIP Projectors for Efficient Intra-modal Alignment](https://mlsanity.com/paper/2603.19862/) (2026)
- [QuARI: Query Adaptive Retrieval Improvement](https://mlsanity.com/paper/2505.21647/) (2025)
- [Infusing fine-grained visual knowledge to Vision-Language Models](https://mlsanity.com/paper/2508.12137/) (2025)
- [One Patch to Caption Them All: A Unified Zero-Shot Captioning Framework](https://mlsanity.com/paper/2510.02898/) (2025)
- [Patch-wise Retrieval: A Bag of Practical Techniques for Instance-level Matching](https://mlsanity.com/paper/2512.12610/) (2025)
- [Find your Needle: Small Object Image Retrieval via Multi-Object Attention Optimization](https://mlsanity.com/paper/2503.07038/) (2025)
- [Instance-Level Generation for Representation Learning](https://mlsanity.com/paper/2510.09171/) (2025)
- [SupScene: Scene-Structured Overlap Supervision for Image Retrieval in Unconstrained SfM](https://mlsanity.com/paper/2601.11930/) (2026)
- [IIR-VLM: In-Context Instance-level Recognition for Large Vision-Language Models](https://mlsanity.com/paper/2601.14188/) (2026)
- [ELViS: Efficient Visual Similarity from Local Descriptors that Generalizes Across Domains](https://mlsanity.com/paper/2603.28603/) (2026)
- [ID-Sim: An Identity-Focused Similarity Metric](https://mlsanity.com/paper/2604.05039/) (2026)
- [Indexing Multimodal Language Models for Large-scale Image Retrieval](https://mlsanity.com/paper/2604.13268/) (2026)
- [Illuminating Visual Identity in Universal Multimodal Embeddings](https://mlsanity.com/paper/2608.01794/) (2026)
- [Class Agnostic Instance-level Descriptor for Visual Instance Search](https://mlsanity.com/paper/2506.16745/) (2025)
- [From Street to Orbit: Training-Free Cross-View Retrieval via Location Semantics and LLM Guidance](https://mlsanity.com/paper/2511.09820/) (2025)
- [Visual Product Search Benchmark](https://mlsanity.com/paper/2603.17186/) (2026)
- [Your CLIP has 164 dimensions of noise: Exploring the embeddings covariance eigenspectrum of contrastively pretrained vision-language transformers](https://mlsanity.com/paper/2605.14893/) (2026)
- [KnifeHunter: Structured Local Representation Learning for Fine-Grained Knife Image Retrieval in Law Enforcement](https://mlsanity.com/paper/2608.07057/) (2026)
- [FoundYou: A Unified Model for Personalized Segmentation and Retrieval](https://mlsanity.com/paper/2608.29917/) (2026)
- [RegRet: Enhancing Region-Level Retrieval in Large Multimodal Models](https://mlsanity.com/paper/2609.16847/) (2026)
- [When Does Geometric View Synthesis Help Wine Label Retrieval? A Public One-Shot Benchmark Across Self-Supervised and Vision-Language Backbones](https://mlsanity.com/paper/2609.33359/) (2026)
- [Masked Swingers: Harnessing Data Augmentation to Advance Autoencoders for Self-Supervised Learning](https://mlsanity.com/paper/2609.38278/) (2026)
- [Learning to Route in Visual Space via Multi-Step Embedding Retrieval](https://mlsanity.com/paper/2609.38743/) (2026)
- [Web Scale on a Budget: Tail-Aware Retrieval Evaluation](https://mlsanity.com/paper/2610.17986/) (2026)

## Abstract (arXiv)

This work introduces ILIAS, a new test dataset for Instance-Level Image retrieval At Scale. It is designed to evaluate the ability of current and future foundation models and retrieval techniques to recognize particular objects. The key benefits over existing datasets include large scale, domain diversity, accurate ground truth, and a performance that is far from saturated. ILIAS includes query and positive images for 1,000 object instances, manually collected to capture challenging conditions and diverse domains. Large-scale retrieval is conducted against 100 million distractor images from YFCC100M. To avoid false negatives without extra annotation effort, we include only query objects confirmed to have emerged after 2014, i.e. the compilation date of YFCC100M. An extensive benchmarking is performed with the following observations: i) models fine-tuned on specific domains, such as landmarks or products, excel in that domain but fail on ILIAS ii) learning a linear adaptation layer using multi-domain class supervision results in performance improvements, especially for vision-language models iii) local descriptors in retrieval re-ranking are still a key ingredient, especially in the presence of severe background clutter iv) the text-to-image performance of the vision-language foundation models is surprisingly close to the corresponding image-to-image case. website: https://vrg.fel.cvut.cz/ilias/