mlsanityData to 7 Oct 2026

The Role of Vocabularies in Learning Sparse Representations for Ranking

arXiv 2509.16621 ↗First posted 20 Sep 2025cs.IR cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Learned Sparse Retrieval (LSR) such as SPLADE has growing interest for effective semantic 1st stage matching while enjoying the efficiency of inverted indices. A recent work on learning SPLADE models with expanded vocabularies (ESPLADE) was proposed to represent queries and documents into a sparse space of custom vocabulary which have different levels of vocabularic granularity. Within this effort, however, there have not been many studies on the role of vocabulary in SPLADE models and their relationship to retrieval efficiency and effectiveness. To study this, we construct BERT models with 100K-sized output vocabularies, one initialized with the ESPLADE pretraining method and one initialized randomly. After finetune on real-world search click logs, we applied logit score-based queries and documents pruning to max size for further balancing efficiency. The experimental result in our evaluation set shows that, when pruning is applied, the two models are effective compared to the 32K-sized normal SPLADE model in the computational budget under the BM25. And the ESPLADE models are more effective than the random vocab model, while having a similar retrieval cost. The result indicates that the size and pretrained weight of output vocabularies play the role of configuring the representational specification for queries, documents, and their interactions in the retrieval engine, beyond their original meaning and purposes in NLP. These findings can provide a new room for improvement for LSR by identifying the importance of representational specification from vocabulary configuration for efficient and effective retrieval.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 34 of its 46 references that are in the map. First posted on arXiv 20 Sep 2025.

Builds on

34 of 46 references.

References · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  6. 2017Billion-scale similarity search with GPUscited by 2,477
  7. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  8. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  9. 2019Passage Re-ranking with BERTcited by 629
  10. 2020Language-agnostic BERT Sentence Embeddingcited by 621
  11. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  12. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 410
22 more
  1. 2016Using the Output Embedding to Improve Language Modelscited by 266
  2. 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
  3. 2019Document Expansion by Query Predictioncited by 225
  4. 2016Transfer Learning for Low-Resource Neural Machine Translationcited by 188
  5. 2020Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillationcited by 143
  6. 2021SPLADE v2: Sparse Lexical and Expansion Model for Information Retrievalcited by 126
  7. 2021COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted Listcited by 113
  8. 2022From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effectivecited by 98
  9. 2021Learning Passage Impacts for Inverted Indexescited by 68
  10. 2019Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrievalcited by 65
  11. 2020Minimizing FLOPs to Learn Efficient Sparse Representationscited by 46
  12. 2020SparTerm: Learning Term-based Sparse Representation for Fast Text Retrievalcited by 45
  13. 2022An Efficiency Study for SPLADE Modelscited by 26
  14. 2023A Unified Framework for Learned Sparse Retrievalcited by 17
  15. 2021Wacky Weights in Learned Sparse Representations and the Revenge of Score-at-a-Time Query Evaluationcited by 15
  16. 2023A Static Pruning Study on Sparse Neural Retrieverscited by 11
  17. 2024Two-Step SPLADE: Simple, Efficient and Effective Approximation of SPLADEcited by 6
  18. 2021Sparsifying Sparse Representations for Passage Retrieval by Top-k Maskingcited by 5
  19. 2023Exploring the Representation Power of SPLADE Modelscited by 5
  20. 2024SPLATE: Sparse Late Interaction Retrievalcited by 4
  21. 2024Improved Learned Sparse Retrieval with Corpus-Specific Vocabulariescited by 3
  22. 2023Representation Sparsification with Hybrid Thresholding for Fast SPLADE-based Document Retrievalcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026From Tokens to Concepts: Leveraging SAE for SPLADEcited by 0
  2. 2026The Pre-Training Study of Expanded-SPLADE Models on Web Document Titlescited by 0
  3. 2026Rescaling MLM-Head for Neural Sparse Retrievalcited by 0
  4. 2026Why Advanced Encoders Lag on Sparse Retrieval? The Answer and an Approach to Bridging Vocabulary Gapscited by 0