# The Role of Vocabularies in Learning Sparse Representations for Ranking

arXiv: https://arxiv.org/abs/2509.16621 · HTML: https://mlsanity.com/paper/2509.16621/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 46 references that are in the map.
- First posted on arXiv 20 Sep 2025.

## Builds on (most cited first, 30 of 34)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval](https://mlsanity.com/paper/2007.00808/) (2020)
- [Passage Re-ranking with BERT](https://mlsanity.com/paper/1901.04085/) (2019)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2010.08191/) (2020)
- [Using the Output Embedding to Improve Language Models](https://mlsanity.com/paper/1608.05859/) (2016)
- [Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling](https://mlsanity.com/paper/2104.06967/) (2021)
- [Document Expansion by Query Prediction](https://mlsanity.com/paper/1904.08375/) (2019)
- [Transfer Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1604.02201/) (2016)
- [Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation](https://mlsanity.com/paper/2010.02666/) (2020)
- [SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval](https://mlsanity.com/paper/2109.10086/) (2021)
- [COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted List](https://mlsanity.com/paper/2104.07186/) (2021)
- [From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective](https://mlsanity.com/paper/2205.04733/) (2022)
- [Learning Passage Impacts for Inverted Indexes](https://mlsanity.com/paper/2104.12016/) (2021)
- [Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrieval](https://mlsanity.com/paper/1910.10687/) (2019)
- [Minimizing FLOPs to Learn Efficient Sparse Representations](https://mlsanity.com/paper/2004.05665/) (2020)
- [SparTerm: Learning Term-based Sparse Representation for Fast Text Retrieval](https://mlsanity.com/paper/2010.00768/) (2020)
- [An Efficiency Study for SPLADE Models](https://mlsanity.com/paper/2207.03834/) (2022)
- [A Unified Framework for Learned Sparse Retrieval](https://mlsanity.com/paper/2303.13416/) (2023)
- [Wacky Weights in Learned Sparse Representations and the Revenge of Score-at-a-Time Query Evaluation](https://mlsanity.com/paper/2110.11540/) (2021)
- [A Static Pruning Study on Sparse Neural Retrievers](https://mlsanity.com/paper/2304.12702/) (2023)
- [Two-Step SPLADE: Simple, Efficient and Effective Approximation of SPLADE](https://mlsanity.com/paper/2404.13357/) (2024)
- [Sparsifying Sparse Representations for Passage Retrieval by Top-k Masking](https://mlsanity.com/paper/2112.09628/) (2021)

## Built on by (most cited first, 4 of 4)

- [From Tokens to Concepts: Leveraging SAE for SPLADE](https://mlsanity.com/paper/2604.21511/) (2026)
- [The Pre-Training Study of Expanded-SPLADE Models on Web Document Titles](https://mlsanity.com/paper/2605.01407/) (2026)
- [Rescaling MLM-Head for Neural Sparse Retrieval](https://mlsanity.com/paper/2606.18811/) (2026)
- [Why Advanced Encoders Lag on Sparse Retrieval? The Answer and an Approach to Bridging Vocabulary Gaps](https://mlsanity.com/paper/2607.00004/) (2026)

## Abstract (arXiv)

Learned Sparse Retrieval (LSR) such as SPLADE has growing interest for effective semantic 1st stage matching while enjoying the efficiency of inverted indices. A recent work on learning SPLADE models with expanded vocabularies (ESPLADE) was proposed to represent queries and documents into a sparse space of custom vocabulary which have different levels of vocabularic granularity. Within this effort, however, there have not been many studies on the role of vocabulary in SPLADE models and their relationship to retrieval efficiency and effectiveness. To study this, we construct BERT models with 100K-sized output vocabularies, one initialized with the ESPLADE pretraining method and one initialized randomly. After finetune on real-world search click logs, we applied logit score-based queries and documents pruning to max size for further balancing efficiency. The experimental result in our evaluation set shows that, when pruning is applied, the two models are effective compared to the 32K-sized normal SPLADE model in the computational budget under the BM25. And the ESPLADE models are more effective than the random vocab model, while having a similar retrieval cost. The result indicates that the size and pretrained weight of output vocabularies play the role of configuring the representational specification for queries, documents, and their interactions in the retrieval engine, beyond their original meaning and purposes in NLP. These findings can provide a new room for improvement for LSR by identifying the importance of representational specification from vocabulary configuration for efficient and effective retrieval.