# STAIR: Learning Sparse Text and Image Representation in Grounded Tokens

arXiv: https://arxiv.org/abs/2301.13081 · HTML: https://mlsanity.com/paper/2301.13081/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 55 references that are in the map.
- First posted on arXiv 30 Jan 2023.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [Remote Sensing Image Scene Classification: Benchmark and State of the Art](https://mlsanity.com/paper/1703.00121/) (2017)
- [Large Batch Optimization for Deep Learning: Training BERT in 76 minutes](https://mlsanity.com/paper/1904.00962/) (2019)
- [SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking](https://mlsanity.com/paper/2107.05720/) (2021)
- [Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologies](https://mlsanity.com/paper/2108.12084/) (2021)
- [SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval](https://mlsanity.com/paper/2109.10086/) (2021)
- [Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrieval](https://mlsanity.com/paper/1910.10687/) (2019)
- [SPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrieval](https://mlsanity.com/paper/2009.13013/) (2020)
- [Minimizing FLOPs to Learn Efficient Sparse Representations](https://mlsanity.com/paper/2004.05665/) (2020)
- [SparTerm: Learning Term-based Sparse Representation for Fast Text Retrieval](https://mlsanity.com/paper/2010.00768/) (2020)
- [Contextualized Query Embeddings for Conversational Search](https://mlsanity.com/paper/2104.08707/) (2021)
- [An Efficiency Study for SPLADE Models](https://mlsanity.com/paper/2207.03834/) (2022)
- [SCANN: Synthesis of Compact and Accurate Neural Networks](https://mlsanity.com/paper/1904.09090/) (2019)
- [Ultra-High Dimensional Sparse Representations with Binarization for Efficient Text Retrieval](https://mlsanity.com/paper/2104.07198/) (2021)

## Built on by (most cited first, 23 of 23)

- [Multimodal Foundation Models: From Specialists to General-Purpose Assistants](https://mlsanity.com/paper/2309.10020/) (2023)
- [Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)](https://mlsanity.com/paper/2402.10376/) (2024)
- [Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2412.02104/) (2024)
- [Less is More: Removing Text-regions Improves CLIP Training Efficiency and Robustness](https://mlsanity.com/paper/2305.05095/) (2023)
- [Multimodal Learned Sparse Retrieval with Probabilistic Expansion Control](https://mlsanity.com/paper/2402.17535/) (2024)
- [Unified Lexical Representation for Interpretable Visual-Language Alignment](https://mlsanity.com/paper/2407.17827/) (2024)
- [MOFI: Learning Image Representations from Noisy Entity Annotated Images](https://mlsanity.com/paper/2306.07952/) (2023)
- [Enhancing Concept Localization in CLIP-based Concept Bottleneck Models](https://mlsanity.com/paper/2510.07115/) (2025)
- [Learning Interpretable Queries for Explainable Image Classification with Information Pursuit](https://mlsanity.com/paper/2312.11548/) (2023)
- [Sparse and Dense Retrievers Learn Better Together: Joint Sparse-Dense Optimization for Text-Image Retrieval](https://mlsanity.com/paper/2508.16707/) (2025)
- [Towards Self-Refinement of Vision-Language Models with Triangular Consistency](https://mlsanity.com/paper/2510.10487/) (2025)
- [Rethinking Sparse Lexical Representations for Image Retrieval in the Age of Rising Multi-Modal Large Language Models](https://mlsanity.com/paper/2408.16296/) (2024)
- [Composed Multi-modal Retrieval: A Survey of Approaches and Applications](https://mlsanity.com/paper/2503.01334/) (2025)
- [GeoMM: On Geodesic Perspective for Multi-modal Learning](https://mlsanity.com/paper/2505.11216/) (2025)
- [Sparse CLIP: Co-Optimizing Interpretability and Performance in Contrastive Learning](https://mlsanity.com/paper/2601.20075/) (2026)
- [UEmbed: Unified Sparse and Dense Multimodal Embeddings](https://mlsanity.com/paper/2608.02583/) (2026)
- [Semantic Token Reweighting for Interpretable and Controllable Text Embeddings in CLIP](https://mlsanity.com/paper/2410.08469/) (2024)
- [Concepts from Representations: Post-hoc Concept Bottleneck Models via Sparse Decomposition of Visual Representations](https://mlsanity.com/paper/2601.12303/) (2026)
- [ICED: Concept-level Machine Unlearning via Interpretable Concept Decomposition](https://mlsanity.com/paper/2605.14309/) (2026)
- [Inference-Free Multimodal Learned Sparse Retrieval for Production-Scale Visual Document Search](https://mlsanity.com/paper/2605.30917/) (2026)
- [Stellar: Scalable Multimodal Document Retrieval for Natural Language Queries](https://mlsanity.com/paper/2606.19960/) (2026)
- [Attributes Should Come from Images, Not Class Names: Distribution-Conditioned Attribute Selection for Vision-Language Models](https://mlsanity.com/paper/2607.18695/) (2026)
- [Concepts Complement Dense Semantics: Learning Compact Sparse Spaces for Text-Image Retrieval](https://mlsanity.com/paper/2609.32671/) (2026)

## Abstract (arXiv)

Image and text retrieval is one of the foundational tasks in the vision and language domain with multiple real-world applications. State-of-the-art approaches, e.g. CLIP, ALIGN, represent images and texts as dense embeddings and calculate the similarity in the dense embedding space as the matching score. On the other hand, sparse semantic features like bag-of-words models are more interpretable, but believed to suffer from inferior accuracy than dense representations. In this work, we show that it is possible to build a sparse semantic representation that is as powerful as, or even better than, dense presentations. We extend the CLIP model and build a sparse text and image representation (STAIR), where the image and text are mapped to a sparse token space. Each token in the space is a (sub-)word in the vocabulary, which is not only interpretable but also easy to integrate with existing information retrieval systems. STAIR model significantly outperforms a CLIP model with +4.9% and +4.3% absolute Recall@1 improvement on COCO-5k textrightarrowimage and imagerightarrowtext retrieval respectively. It also achieved better performance on both of ImageNet zero-shot and linear probing compared to CLIP.