# Minimizing FLOPs to Learn Efficient Sparse Representations

arXiv: https://arxiv.org/abs/2004.05665 · HTML: https://mlsanity.com/paper/2004.05665/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Apr 2020.

## Built on by (most cited first, 30 of 46)

- [SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval](https://mlsanity.com/paper/2109.10086/) (2021)
- [From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective](https://mlsanity.com/paper/2205.04733/) (2022)
- [Controlled Text Generation as Continuous Optimization with Multiple Constraints](https://mlsanity.com/paper/2108.01850/) (2021)
- [Gradient-Based Constrained Sampling from Language Models](https://mlsanity.com/paper/2205.12558/) (2022)
- [WebBrain: Learning to Generate Factually Correct Articles for Queries by Grounding on Large Web Corpus](https://mlsanity.com/paper/2304.04358/) (2023)
- [Learning strides in convolutional neural networks](https://mlsanity.com/paper/2202.01653/) (2022)
- [An Efficiency Study for SPLADE Models](https://mlsanity.com/paper/2207.03834/) (2022)
- [STAIR: Learning Sparse Text and Image Representation in Grounded Tokens](https://mlsanity.com/paper/2301.13081/) (2023)
- [UnifieR: A Unified Retriever for Large-Scale Retrieval](https://mlsanity.com/paper/2205.11194/) (2022)
- [Planning Ahead in Generative Retrieval: Guiding Autoregressive Generation through Simultaneous Decoding](https://mlsanity.com/paper/2404.14600/) (2024)
- [CSPLADE: Learned Sparse Retrieval with Causal Language Models](https://mlsanity.com/paper/2504.10816/) (2025)
- [Unified Lexical Representation for Interpretable Visual-Language Alignment](https://mlsanity.com/paper/2407.17827/) (2024)
- [DiSCo: LLM Knowledge Distillation for Efficient Sparse Retrieval in Conversational Search](https://mlsanity.com/paper/2410.14609/) (2024)
- [Mistral-SPLADE: LLMs for better Learned Sparse Retrieval](https://mlsanity.com/paper/2408.11119/) (2024)
- [Towards Competitive Search Relevance For Inference-Free Learned Sparse Retrievers](https://mlsanity.com/paper/2411.04403/) (2024)
- [Granite Embedding Models](https://mlsanity.com/paper/2502.20204/) (2025)
- [Ultra-High Dimensional Sparse Representations with Binarization for Efficient Text Retrieval](https://mlsanity.com/paper/2104.07198/) (2021)
- [Learning Retrieval Models with Sparse Autoencoders](https://mlsanity.com/paper/2603.13277/) (2026)
- [Sparsifying Sparse Representations for Passage Retrieval by Top-k Masking](https://mlsanity.com/paper/2112.09628/) (2021)
- [LACONIC: Dense-Level Effectiveness for Scalable Sparse Retrieval via a Two-Phase Training Curriculum](https://mlsanity.com/paper/2601.01684/) (2026)
- [Composite Code Sparse Autoencoders for first stage retrieval](https://mlsanity.com/paper/2204.07023/) (2022)
- [The Role of Vocabularies in Learning Sparse Representations for Ranking](https://mlsanity.com/paper/2509.16621/) (2025)
- [An Experimental Study on Pretraining Transformers from Scratch for IR](https://mlsanity.com/paper/2301.10444/) (2023)
- [Parameter-Efficient Sparse Retrievers and Rerankers using Adapters](https://mlsanity.com/paper/2303.13220/) (2023)
- [Asymmetrical estimator for training encapsulated deep photonic neural networks](https://mlsanity.com/paper/2405.18458/) (2024)
- [An Alternative to FLOPS Regularization to Effectively Productionize SPLADE-Doc](https://mlsanity.com/paper/2505.15070/) (2025)
- [Unsupervised Domain Adaptation for Sparse Retrieval by Filling Vocabulary and Word Frequency Gaps](https://mlsanity.com/paper/2211.03988/) (2022)
- [SPRINT: A Unified Toolkit for Evaluating and Demystifying Zero-shot Neural Sparse Retrieval](https://mlsanity.com/paper/2307.10488/) (2023)
- [On the Challenges and Opportunities of Learned Sparse Retrieval for Code](https://mlsanity.com/paper/2603.22008/) (2026)
- [UEmbed: Unified Sparse and Dense Multimodal Embeddings](https://mlsanity.com/paper/2608.02583/) (2026)

## Abstract (arXiv)

Deep representation learning has become one of the most widely adopted approaches for visual search, recommendation, and identification. Retrieval of such representations from a large database is however computationally challenging. Approximate methods based on learning compact representations, have been widely explored for this problem, such as locality sensitive hashing, product quantization, and PCA. In this work, in contrast to learning compact representations, we propose to learn high dimensional and sparse representations that have similar representational capacity as dense embeddings while being more efficient due to sparse matrix multiplication operations which can be much faster than dense multiplication. Following the key insight that the number of operations decreases quadratically with the sparsity of embeddings provided the non-zero entries are distributed uniformly across dimensions, we propose a novel approach to learn such distributed sparse embeddings via the use of a carefully constructed regularization function that directly minimizes a continuous relaxation of the number of floating-point operations (FLOPs) incurred during retrieval. Our experiments show that our approach is competitive to the other baselines and yields a similar or better speed-vs-accuracy tradeoff on practical datasets.