Rethinking Sparse Lexical Representations for Image Retrieval in the Age of Rising Multi-Modal Large Language Models
On the map: LLM reasoning → Multimodal reasoning
Abstract (arXiv)
In this paper, we rethink sparse lexical representations for image retrieval. By utilizing multi-modal large language models (M-LLMs) that support visual prompting, we can extract image features and convert them into textual data, enabling us to utilize efficient sparse retrieval algorithms employed in natural language processing for image retrieval tasks. To assist the LLM in extracting image features, we apply data augmentation techniques for key expansion and analyze the impact with a metric for relevance between images and textual data. We empirically show the superior precision and recall performance of our image retrieval method compared to conventional vision-language model-based methods on the MS-COCO, PASCAL VOC, and NUS-WIDE datasets in a keyword-based image retrieval scenario, where keywords serve as search queries. We also demonstrate that the retrieval performance can be improved by iteratively incorporating keywords into search queries.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 45 of its 64 references that are in the map. First posted on arXiv 29 Aug 2024.
Builds on
45 of 64 references.
References · 45
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2023GPT-4 Technical Reportcited by 18,132
- 2014Microsoft COCO: Common Objects in Contextcited by 17,498
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
- 2023Visual Instruction Tuningcited by 7,792
- 2017Mask R-CNNcited by 7,259
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
- 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
- 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
33 more
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
- 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
- 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
- 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
- 2018YOLOv3: An Incremental Improvementcited by 2,652
- 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
- 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
- 2015SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentationcited by 1,483
- 2015Spatial Transformer Networkscited by 1,423
- 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
- 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
- 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
- 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
- 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
- 2019Cross-Modal Self-Attention Network for Referring Image Segmentationcited by 263
- 2022BlenderBot 3: a deployed conversational agent that continually learns to responsibly engagecited by 197
- 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 185
- 2024BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrievalcited by 153
- 2016Deep Image Retrieval: Learning global representations for image searchcited by 147
- 2021Simple Entity-Centric Questions Challenge Dense Retrieverscited by 140
- 2020ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Datacited by 136
- 2023Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluationcited by 78
- 2023Toward Verifiable and Reproducible Human Evaluation for Text-to-Image Generationcited by 66
- 2021Instance-level Image Retrieval using Reranking Transformerscited by 63
- 2016Deep Cross-Modal Hashingcited by 54
- 2023Towards Language Models That Can See: Computer Vision Through the LENS of Natural Languagecited by 53
- 2019Reflective Decoding Network for Image Captioningcited by 31
- 2023STAIR: Learning Sparse Text and Image Representation in Grounded Tokenscited by 23
- 2023TagGPT: Large Language Models are Zero-shot Multimodal Taggerscited by 15
- 2022Retrieval-based Disentangled Representation Learning with Natural Language Supervisioncited by 5
- 2018Semantic Image Retrieval by Uniting Deep Neural Networks and Cognitive Architecturescited by 2
- 2024Multimodal Learned Sparse Retrieval for Image Suggestioncited by 2
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Concepts Complement Dense Semantics: Learning Compact Sparse Spaces for Text-Image Retrieval
- 2025Sparse and Dense Retrievers Learn Better Together: Joint Sparse-Dense Optimization for Text-Image Retrieval
- 2026Unified Multimodal and Multilingual Retrieval via Multi-Task Learning with NLU Integration
- 2026Indexing Multimodal Language Models for Large-scale Image Retrieval
- 2023MLLMs-Augmented Visual-Language Representation Learning
- 2025Category-level Text-to-Image Retrieval Improved: Bridging the Domain Gap with Diffusion Models and Vision Encoders
- 2025A Little More Like This: Text-to-Image Retrieval with Vision-Language Models Using Relevance Feedback
- 2022HiVLP: Hierarchical Vision-Language Pre-Training for Fast Image-Text Retrieval