mlsanityData to 14 Oct 2026

From Tokens to Concepts: Leveraging SAE for SPLADE

arXiv 2604.21511 ↗First posted 23 Apr 2026cs.IR cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Learned Sparse IR models, such as SPLADE, offer an excellent efficiency-effectiveness tradeoff. However, they rely on the underlying backbone vocabulary, which might hinder performance (polysemicity and synonymy) and pose a challenge for multi-lingual and multi-modal usages. To solve this limitation, we propose to replace the backbone vocabulary with a latent space of semantic concepts learned using Sparse Auto-Encoders (SAE). Throughout this paper, we study the compatibility of these 2 concepts, explore training approaches, and analyze the differences between our SAE-SPLADE model and traditional SPLADE models. Our experiments demonstrate that SAE-SPLADE achieves retrieval performance comparable to SPLADE on both in-domain and out-of-domain tasks while offering improved efficiency.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 33 of its 53 references that are in the map. First posted on arXiv 23 Apr 2026.

Builds on

33 of 53 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  7. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  8. 2020Overview of the TREC 2019 deep learning trackcited by 286
  9. 2021SPLADE: Sparse Lexical and Expansion Model for First Stage Rankingcited by 286
  10. 2021Overview of the TREC 2020 deep learning trackcited by 196
  11. 2025Learning Multi-Level Features with Matryoshka Sparse Autoencoderscited by 143
  12. 2021mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Datasetcited by 116
21 more
  1. 2021COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted Listcited by 113
  2. 2022PLAID: An Efficient Engine for Late Interaction Retrievalcited by 101
  3. 2022From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effectivecited by 98
  4. 2022SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrievalcited by 90
  5. 2024Anisotropy Is Inherent to Self-Attention in Transformerscited by 63
  6. 2020Minimizing FLOPs to Learn Efficient Sparse Representationscited by 46
  7. 2022Curriculum Learning for Dense Retrieval Distillationcited by 38
  8. 2020Expansion via Prediction of Importance with Contextualizationcited by 26
  9. 2025Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representationcited by 26
  10. 2023A Unified Framework for Learned Sparse Retrievalcited by 17
  11. 2024Decoding Dark Matter: Specialized Sparse Autoencoders for Interpreting Rare Concepts in Foundation Modelscited by 16
  12. 2024Faster Learned Sparse Retrieval with Block-Max Pruningcited by 10
  13. 2024Interpret and Control Dense Retrieval with Sparse Latent Featurescited by 10
  14. 2024DyVo: Dynamic Vocabularies for Learned Sparse Retrieval with Entitiescited by 8
  15. 2025Decoding Dense Embeddings: Sparse Autoencoders for Interpreting and Discretizing Dense Retrievalcited by 8
  16. 2021Ultra-High Dimensional Sparse Representations with Binarization for Efficient Text Retrievalcited by 7
  17. 2026Learning Retrieval Models with Sparse Autoencoderscited by 6
  18. 2026LACONIC: Dense-Level Effectiveness for Scalable Sparse Retrieval via a Two-Phase Training Curriculumcited by 5
  19. 2022Composite Code Sparse Autoencoders for first stage retrievalcited by 4
  20. 2025The Role of Vocabularies in Learning Sparse Representations for Rankingcited by 4
  21. 2023Extending English IR methods to multi-lingual IRcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Disentangling Dense Embeddings with Sparse Autoencoders
  2. 2025Incorporating Hierarchical Semantics in Sparse Autoencoder Architectures
  3. 2025AbsTopK: Rethinking Sparse Autoencoders For Bidirectional Features
  4. 2025Temporal Sparse Autoencoders: Leveraging the Sequential Nature of Language for Interpretability
  5. 2026Sparse Autoencoders for Mechanistic Interpretability in NLP: A Critical Survey of Recoverability, Interpretability, and Utility
  6. 2025Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
  7. 2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models
  8. 2026Beyond Token Scale: Chunk-Level Sparse Autoencoders for Reliable Semantic Feature Discovery