mlsanityData to 7 Oct 2026

Learning Retrieval Models with Sparse Autoencoders

arXiv 2603.13277 ↗First posted 27 Feb 2026cs.LG cs.AI cs.IRcited by 5

Not on the map. Why?

Abstract (arXiv)

Sparse autoencoders (SAEs) provide a powerful mechanism for decomposing the dense representations produced by Large Language Models (LLMs) into interpretable latent features. We posit that SAEs constitute a natural foundation for Learned Sparse Retrieval (LSR), whose objective is to encode queries and documents into high-dimensional sparse representations optimized for efficient retrieval. In contrast to existing LSR approaches that project input sequences into the vocabulary space, SAE-based representations offer the potential to produce more semantically structured, expressive, and language-agnostic features. Building on this insight, we introduce SPLARE, a method to train SAE-based LSR models. Our experiments, relying on recently released open-source SAEs, demonstrate that this technique consistently outperforms vocabulary-based LSR in multilingual and out-of-domain settings. SPLARE-7B, a multilingual retrieval model capable of producing generalizable sparse latent embeddings for a wide range of languages and domains, achieves top results on MMTEB's multilingual and English retrieval tasks. We also developed a 2B-parameter variant with a significantly lighter footprint.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 68 of its 91 references that are in the map. First posted on arXiv 27 Feb 2026.

Builds on

68 of 91 references.

References · 68

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  9. 2017Billion-scale similarity search with GPUscited by 2,477
  10. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  11. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  12. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
38 more
  1. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  2. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  3. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  4. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  5. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  6. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  7. 2019Passage Re-ranking with BERTcited by 629
  8. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  9. 2024Scaling and evaluating sparse autoencoderscited by 530
  10. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 466
  11. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 396
  12. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 381
  13. 2024Multilingual E5 Text Embeddings: A Technical Reportcited by 357
  14. 2023Improving Text Embeddings with Large Language Modelscited by 295
  15. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 293
  16. 2021SPLADE: Sparse Lexical and Expansion Model for First Stage Rankingcited by 283
  17. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoderscited by 283
  18. 2022Matryoshka Representation Learningcited by 268
  19. 2022Promptagator: Few-shot Dense Retrieval From 8 Examplescited by 233
  20. 2021Overview of the TREC 2020 deep learning trackcited by 193
  21. 2024ColPali: Efficient Document Retrieval with Vision Language Modelscited by 190
  22. 2020Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillationcited by 143
  23. 2024Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoderscited by 133
  24. 2021Mr. TyDi: A Multi-lingual Benchmark for Dense Retrievalcited by 118
  25. 2025Are Sparse Autoencoders Useful? A Case Study in Sparse Probingcited by 109
  26. 2022From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effectivecited by 98
  27. 2025Gemini Embedding: Generalizable Embeddings from Geminicited by 93
  28. 2024NV-Retriever: Improving text embedding models with effective hard-negative miningcited by 83
  29. 2024SPLADE-v3: New baselines for SPLADEcited by 69
  30. 2021Learning Passage Impacts for Inverted Indexescited by 68
  31. 2020Distilling Dense Representations for Ranking using Tightly-Coupled Teacherscited by 52
  32. 2020Minimizing FLOPs to Learn Efficient Sparse Representationscited by 46
  33. 2023XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languagescited by 34
  34. 2025Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representationcited by 25
  35. 2025Llama Nemoretriever Colembed: Top-Performing Text-Image Retrieval Modelcited by 24
  36. 2024Efficient Inverted Indexes for Approximate Retrieval over Learned Sparse Representationscited by 23
  37. 2024A Thorough Comparison of Cross-Encoders and LLMs for Reranking SPLADEcited by 18
  38. 2024Multimodal Learned Sparse Retrieval with Probabilistic Expansion Controlcited by 12

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Visual Words Meet BM25: Sparse Auto-Encoder Visual Word Scoring for Image Retrievalcited by 2
  2. 2026On the Challenges and Opportunities of Learned Sparse Retrieval for Codecited by 2
  3. 2026UEmbed: Unified Sparse and Dense Multimodal Embeddingscited by 2
  4. 2026From Tokens to Concepts: Leveraging SAE for SPLADEcited by 0
  5. 2026Latent Terms: Dense Retrievers Contain Trivially Extractable BM25-ready Zipfian Vocabulariescited by 0