# Nearest Neighbor Normalization Improves Multimodal Retrieval

arXiv: https://arxiv.org/abs/2410.24114 · HTML: https://mlsanity.com/paper/2410.24114/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 21 references that are in the map.
- First posted on arXiv 31 Oct 2024.

## Builds on (most cited first, 16 of 16)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020)
- [Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts](https://mlsanity.com/paper/2111.08276/) (2021)
- [A Prompt Array Keeps the Bias Away: Debiasing Vision-Language Models with Adversarial Learning](https://mlsanity.com/paper/2203.11933/) (2022)
- [Are Gender-Neutral Queries Really Gender-Neutral? Mitigating Gender Bias in Image Search](https://mlsanity.com/paper/2109.05433/) (2021)
- [Cross Modal Retrieval with Querybank Normalisation](https://mlsanity.com/paper/2112.12777/) (2021)
- [AD-CLIP: Adapting Domains in Prompt Space Using CLIP](https://mlsanity.com/paper/2308.05659/) (2023)
- [VisoGender: A dataset for benchmarking gender bias in image-text pronoun resolution](https://mlsanity.com/paper/2306.12424/) (2023)
- [Retrieval-Enhanced Contrastive Vision-Text Models](https://mlsanity.com/paper/2306.07196/) (2023)
- [FairCLIP: Social Bias Elimination based on Attribute Prototype Learning and Representation Neutralization](https://mlsanity.com/paper/2210.14562/) (2022)
- [Balance Act: Mitigating Hubness in Cross-Modal Retrieval with Query and Gallery Banks](https://mlsanity.com/paper/2310.11612/) (2023)

## Built on by (most cited first, 7 of 7)

- [SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers](https://mlsanity.com/paper/2504.00255/) (2025)
- [NeighborRetr: Balancing Hub Centrality in Cross-Modal Retrieval](https://mlsanity.com/paper/2503.10526/) (2025)
- [Prediction hubs are context-informed frequent tokens in LLMs](https://mlsanity.com/paper/2502.10201/) (2025)
- [One Single Hub Text Breaks CLIP: Identifying Vulnerabilities in Cross-Modal Encoders via Hubness](https://mlsanity.com/paper/2604.27674/) (2026)
- [Hacking Neural Evaluation Metrics with Single Hub Text](https://mlsanity.com/paper/2512.16323/) (2025)
- [Unified Multi-Dataset Training for TBPS](https://mlsanity.com/paper/2601.14978/) (2026)
- [MINER: Multi-crop INference-time Enhancement for Rare-Object Retrieval with Frozen Dual Encoders](https://mlsanity.com/paper/2609.27142/) (2026)

## Abstract (arXiv)

Multimodal models leverage large-scale pre-training to achieve strong but still imperfect performance on tasks such as image captioning, visual question answering, and cross-modal retrieval. In this paper, we present a simple and efficient method for correcting errors in trained contrastive image-text retrieval models with no additional training, called Nearest Neighbor Normalization (NNN). We show an improvement on retrieval metrics in both text retrieval and image retrieval for all of the contrastive models that we tested (CLIP, BLIP, ALBEF, SigLIP, BEiT) and for both of the datasets that we used (MS-COCO and Flickr30k). NNN requires a reference database, but does not require any training on this database, and can even increase the retrieval accuracy of a model after finetuning.