mlsanityData to 7 Oct 2026

ELIP: Enhanced Visual-Language Foundation Models for Image Retrieval

arXiv 2502.15682 ↗First posted 21 Feb 2025cs.CVcited by 9

Not on the map. Why?

Abstract (arXiv)

The objective in this paper is to improve the performance of text-to-image retrieval. To this end, we introduce a new framework that can boost the performance of large-scale pre-trained vision-language models, so that they can be used for text-to-image re-ranking. The approach, Enhanced Language-Image Pre-training (ELIP), uses the text query, via a simple MLP mapping network, to predict a set of visual prompts to condition the ViT image encoding. ELIP can easily be applied to the commonly used CLIP, SigLIP and BLIP-2 networks. To train the architecture with limited computing resources, we develop a 'student friendly' best practice, involving global hard sample mining, and curation of a large-scale dataset. On the evaluation side, we set up two new out-of-distribution (OOD) benchmarks, Occluded COCO and ImageNet-R, to assess the zero-shot generalisation of the models to different domains. The results demonstrate that ELIP significantly boosts CLIP/SigLIP/SigLIP-2 text-to-image retrieval performance and outperforms BLIP-2 on several benchmarks, as well as providing an easy means to adapt to OOD datasets.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 83 of its 119 references that are in the map. First posted on arXiv 21 Feb 2025.

Builds on

83 of 119 references.

References · 83

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  12. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
38 more
  1. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  2. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  3. 2022Visual Prompt Tuningcited by 1,674
  4. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  5. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  6. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  7. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  8. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
  9. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  10. 2020Contrastive Learning with Hard Negative Samplescited by 619
  11. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  12. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  13. 2018Stacked Cross Attention for Image-Text Matchingcited by 414
  14. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  15. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 316
  16. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 295
  17. 2017Sampling Matters in Deep Embedding Learningcited by 285
  18. 2023Data Filtering Networkscited by 247
  19. 2021Patch-NetVLAD: Multi-Scale Fusion of Locally-Global Descriptors for Place Recognitioncited by 236
  20. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 233
  21. 2017Working hard to know your neighbor's margins: Local descriptor learning losscited by 212
  22. 2019Visual Semantic Reasoning for Image-Text Matchingcited by 185
  23. 2023Zero-Shot Composed Image Retrieval with Textual Inversioncited by 171
  24. 2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 165
  25. 2021Probabilistic Embeddings for Cross-Modal Retrievalcited by 150
  26. 2020Learning the Best Pooling Strategy for Visual Semantic Embeddingcited by 136
  27. 2024Rho-1: Not All Tokens Are What You Needcited by 132
  28. 2015Online Batch Selection for Faster Training of Neural Networkscited by 121
  29. 2021Similarity Reasoning and Filtration for Image-Text Matchingcited by 121
  30. 2017Learning Two-Branch Neural Networks for Image-Text Matching Taskscited by 119
  31. 2020IMRAM: Iterative Matching with Recurrent Attention Memory for Cross-Modal Image-Text Retrievalcited by 115
  32. 2019CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrievalcited by 110
  33. 2024LamRA: Large Multimodal Model as Your Advanced Retrieval Assistantcited by 106
  34. 2014Deep Fragment Embeddings for Bidirectional Image Sentence Mappingcited by 99
  35. 2017Smart Mining for Deep Metric Learningcited by 96
  36. 2023MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Trainingcited by 96
  37. 2020Graph Structured Network for Image-Text Matchingcited by 93
  38. 2021Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformerscited by 93

50 most cited of 83 All 83 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 9.

9 papers.

  1. 2026Steerable Visual Representationscited by 4
  2. 2026EGM: Efficient Visual Grounding Language Modelscited by 2
  3. 2026FlowCIR: Semantic Transport via Flow Matching for Zero-Shot Composed Image Retrievalcited by 1
  4. 2026CMI-Mem: Toward Generalizable Long-Term Memory Management via CMI-Augmented Reinforcement Learningcited by 1
  5. 2026ViCLIP-OT: The First Foundation Vision-Language Model for Vietnamese Image-Text Retrieval with Optimal Transportcited by 0
  6. 2026Robust Fuzzy Multi-view Learning under View Conflictcited by 0
  7. 2026LARE: Low-Attention Region Encoding for Text-Image Retrievalcited by 0
  8. 2026CloSeR: Unified Relational Distillation from Closed-Set Teachers for Category Discoverycited by 0
  9. 2026MINER: Multi-crop INference-time Enhancement for Rare-Object Retrieval with Frozen Dual Encoderscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023ELIP: Efficient Discriminative Language-Image Pre-training with Fewer Vision Tokens
  2. 2022HiVLP: Hierarchical Vision-Language Pre-Training for Fast Image-Text Retrieval
  3. 2026Few Shots Text to Image Retrieval: New Benchmarking Dataset and Optimization Methods
  4. 2026Seeing Through Words: Controlling Visual Retrieval Quality with Language Models
  5. 2024Enhancing Interactive Image Retrieval With Query Rewriting Using Large Language Models and Vision Language Models
  6. 2024Rethinking Sparse Lexical Representations for Image Retrieval in the Age of Rising Multi-Modal Large Language Models
  7. 2023Revisiting the Role of Language Priors in Vision-Language Models
  8. 2024Adapting Dual-encoder Vision-language Models for Paraphrased Retrieval