# SparTerm: Learning Term-based Sparse Representation for Fast Text Retrieval

arXiv: https://arxiv.org/abs/2010.00768 · HTML: https://mlsanity.com/paper/2010.00768/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Oct 2020.

## Built on by (most cited first, 30 of 45)

- [SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval](https://mlsanity.com/paper/2109.10086/) (2021)
- [Autoregressive Search Engines: Generating Substrings as Document Identifiers](https://mlsanity.com/paper/2204.10628/) (2022)
- [What Are You Token About? Dense Retrieval as Distributions Over the Vocabulary](https://mlsanity.com/paper/2212.10380/) (2022)
- [An Efficiency Study for SPLADE Models](https://mlsanity.com/paper/2207.03834/) (2022)
- [A Proposed Conceptual Framework for a Representational Approach to Information Retrieval](https://mlsanity.com/paper/2110.01529/) (2021)
- [STAIR: Learning Sparse Text and Image Representation in Grounded Tokens](https://mlsanity.com/paper/2301.13081/) (2023)
- [In Defense of Cross-Encoders for Zero-Shot Retrieval](https://mlsanity.com/paper/2212.06121/) (2022)
- [Utilizing BERT for Information Retrieval: Survey, Applications, Resources, and Challenges](https://mlsanity.com/paper/2403.00784/) (2024)
- [CITADEL: Conditional Token Interaction via Dynamic Lexical Routing for Efficient and Effective Multi-Vector Retrieval](https://mlsanity.com/paper/2211.10411/) (2022)
- [Resources for Brewing BEIR: Reproducible Reference Models and an Official Leaderboard](https://mlsanity.com/paper/2306.07471/) (2023)
- [Interpretable Text Embeddings and Text Similarity Explanation: A Survey](https://mlsanity.com/paper/2502.14862/) (2025)
- [CSPLADE: Learned Sparse Retrieval with Causal Language Models](https://mlsanity.com/paper/2504.10816/) (2025)
- [Query-as-context Pre-training for Dense Passage Retrieval](https://mlsanity.com/paper/2212.09598/) (2022)
- [Towards Competitive Search Relevance For Inference-Free Learned Sparse Retrievers](https://mlsanity.com/paper/2411.04403/) (2024)
- [Granite Embedding Models](https://mlsanity.com/paper/2502.20204/) (2025)
- [Ultra-High Dimensional Sparse Representations with Binarization for Efficient Text Retrieval](https://mlsanity.com/paper/2104.07198/) (2021)
- [Breaking the Hourglass Phenomenon of Residual Quantization: Enhancing the Upper Bound of Generative Retrieval](https://mlsanity.com/paper/2407.21488/) (2024)
- [CAME: Competitively Learning a Mixture-of-Experts Model for First-stage Retrieval](https://mlsanity.com/paper/2311.02834/) (2023)
- [Sparsifying Sparse Representations for Passage Retrieval by Top-k Masking](https://mlsanity.com/paper/2112.09628/) (2021)
- [Retrieval-based Disentangled Representation Learning with Natural Language Supervision](https://mlsanity.com/paper/2212.07699/) (2022)
- [CoT-MAE v2: Contextual Masked Auto-Encoder with Multi-view Modeling for Passage Retrieval](https://mlsanity.com/paper/2304.03158/) (2023)
- [LACONIC: Dense-Level Effectiveness for Scalable Sparse Retrieval via a Two-Phase Training Curriculum](https://mlsanity.com/paper/2601.01684/) (2026)
- [Composite Code Sparse Autoencoders for first stage retrieval](https://mlsanity.com/paper/2204.07023/) (2022)
- [Deep Bag-of-Words Model: An Efficient and Interpretable Relevance Architecture for Chinese E-Commerce](https://mlsanity.com/paper/2407.09395/) (2024)
- [The Role of Vocabularies in Learning Sparse Representations for Ranking](https://mlsanity.com/paper/2509.16621/) (2025)
- [Pan More Gold from the Sand: Refining Open-domain Dialogue Training with Noisy Self-Retrieval Generation](https://mlsanity.com/paper/2201.11367/) (2022)
- [SPRINT: A Unified Toolkit for Evaluating and Demystifying Zero-shot Neural Sparse Retrieval](https://mlsanity.com/paper/2307.10488/) (2023)
- [Efficient and Interpretable Information Retrieval for Product Question Answering with Heterogeneous Data](https://mlsanity.com/paper/2405.13173/) (2024)
- [Fine-Tuning Large Language Models and Evaluating Retrieval Methods for Improved Question Answering on Building Codes](https://mlsanity.com/paper/2505.04666/) (2025)
- [Query Expansion and Entity Weighting for Query Reformulation Retrieval in Voice Assistant Systems](https://mlsanity.com/paper/2202.13869/) (2022)

## Abstract (arXiv)

Term-based sparse representations dominate the first-stage text retrieval in industrial applications, due to its advantage in efficiency, interpretability, and exact term matching. In this paper, we study the problem of transferring the deep knowledge of the pre-trained language model (PLM) to Term-based Sparse representations, aiming to improve the representation capacity of bag-of-words(BoW) method for semantic-level matching, while still keeping its advantages. Specifically, we propose a novel framework SparTerm to directly learn sparse text representations in the full vocabulary space. The proposed SparTerm comprises an importance predictor to predict the importance for each term in the vocabulary, and a gating controller to control the term activation. These two modules cooperatively ensure the sparsity and flexibility of the final text representation, which unifies the term-weighting and expansion in the same framework. Evaluated on MSMARCO dataset, SparTerm significantly outperforms traditional sparse methods and achieves state of the art ranking performance among all the PLM-based sparse models.