mlsanityData to 14 Oct 2026

Nearest Neighbor Normalization Improves Multimodal Retrieval

arXiv 2410.24114 ↗First posted 31 Oct 2024cs.CV cs.AI cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

Multimodal models leverage large-scale pre-training to achieve strong but still imperfect performance on tasks such as image captioning, visual question answering, and cross-modal retrieval. In this paper, we present a simple and efficient method for correcting errors in trained contrastive image-text retrieval models with no additional training, called Nearest Neighbor Normalization (NNN). We show an improvement on retrieval metrics in both text retrieval and image retrieval for all of the contrastive models that we tested (CLIP, BLIP, ALBEF, SigLIP, BEiT) and for both of the datasets that we used (MS-COCO and Flickr30k). NNN requires a reference database, but does not require any training on this database, and can even increase the retrieval accuracy of a model after finetuning.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 16 of its 21 references that are in the map. First posted on arXiv 31 Oct 2024.

Builds on

16 of 21 references.

References · 16

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  6. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  7. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 641
  8. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 256
  9. 2022A Prompt Array Keeps the Bias Away: Debiasing Vision-Language Models with Adversarial Learningcited by 104
  10. 2021Are Gender-Neutral Queries Really Gender-Neutral? Mitigating Gender Bias in Image Searchcited by 90
  11. 2021Cross Modal Retrieval with Querybank Normalisationcited by 72
  12. 2023AD-CLIP: Adapting Domains in Prompt Space Using CLIPcited by 62
4 more
  1. 2023VisoGender: A dataset for benchmarking gender bias in image-text pronoun resolutioncited by 52
  2. 2023Retrieval-Enhanced Contrastive Vision-Text Modelscited by 36
  3. 2022FairCLIP: Social Bias Elimination based on Attribute Prototype Learning and Representation Neutralizationcited by 20
  4. 2023Balance Act: Mitigating Hubness in Cross-Modal Retrieval with Query and Gallery Bankscited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 3.

7 papers.

  1. 2025SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Paperscited by 43
  2. 2025NeighborRetr: Balancing Hub Centrality in Cross-Modal Retrievalcited by 6
  3. 2025Prediction hubs are context-informed frequent tokens in LLMscited by 3
  4. 2026One Single Hub Text Breaks CLIP: Identifying Vulnerabilities in Cross-Modal Encoders via Hubnesscited by 2
  5. 2025Hacking Neural Evaluation Metrics with Single Hub Textcited by 1
  6. 2026Unified Multi-Dataset Training for TBPScited by 0
  7. 2026MINER: Multi-crop INference-time Enhancement for Rare-Object Retrieval with Frozen Dual Encoderscited by 0