mlsanityData to 14 Oct 2026

Decoding Dense Embeddings: Sparse Autoencoders for Interpreting and Discretizing Dense Retrieval

arXiv 2506.00041 ↗First posted 28 May 2025cs.IR cs.LGcited by 8

On the map: Mechanistic interpretability → SAE features and their applications

Abstract (arXiv)

Despite their strong performance, Dense Passage Retrieval (DPR) models suffer from a lack of interpretability. In this work, we propose a novel interpretability framework that leverages Sparse Autoencoders (SAEs) to decompose previously uninterpretable dense embeddings from DPR models into distinct, interpretable latent concepts. We generate natural language descriptions for each latent concept, enabling human interpretations of both the dense embeddings and the query-document similarity scores of DPR models. We further introduce Concept-Level Sparse Retrieval (CL-SR), a retrieval framework that directly utilizes the extracted latent concepts as indexing units. CL-SR effectively combines the semantic expressiveness of dense embeddings with the transparency and efficiency of sparse representations. We show that CL-SR achieves high index-space and computational efficiency while maintaining robust performance across vocabulary and semantic mismatches.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability. Builds on 21 of its 45 references that are in the map. First posted on arXiv 28 May 2025.

Builds on

21 of 45 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders19 Jul
  2. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Models15 Sep

Other references · 19

  1. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  2. 2021Large Dual Encoders Are Generalizable Retrieverscited by 456
  3. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 435
  4. 2020Overview of the TREC 2019 deep learning trackcited by 286
  5. 2021SPLADE: Sparse Lexical and Expansion Model for First Stage Rankingcited by 286
  6. 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
  7. 2021Overview of the TREC 2020 deep learning trackcited by 196
  8. 2021COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted Listcited by 113
  9. 2022SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrievalcited by 90
  10. 2025Sparse Autoencoders Do Not Find Canonical Units of Analysiscited by 88
  11. 2024SPLADE-v3: New baselines for SPLADEcited by 72
  12. 2021A Few Brief Notes on DeepImpact, COIL, and a Conceptual Framework for Information Retrieval Techniquescited by 69
7 more
  1. 2021Learning Passage Impacts for Inverted Indexescited by 68
  2. 2024BMRetriever: Tuning Large Language Models as Better Biomedical Text Retrieverscited by 38
  3. 2022CITADEL: Conditional Token Interaction via Dynamic Lexical Routing for Efficient and Effective Multi-Vector Retrievalcited by 19
  4. 2024Interpret and Control Dense Retrieval with Sparse Latent Featurescited by 10
  5. 2021Towards Axiomatic Explanations for Neural Ranking Modelscited by 7
  6. 2023Explain like I am BM25: Interpreting a Dense Model's Ranked-List with a Sparse Approximationcited by 5
  7. 2021Interpreting Dense Retrieval as Mixture of Topicscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 7.

8 papers.

  1. 2026Learning Retrieval Models with Sparse Autoencoderscited by 6
  2. 2026Visual Words Meet BM25: Sparse Auto-Encoder Visual Word Scoring for Image Retrievalcited by 2
  3. 2025QueStER: Query Specification for Generative keyword-based Retrievalcited by 1
  4. 2026Revolutionizing Long-Term Memory in AI: New Horizons with High-Capacity and High-Speed Storagecited by 1
  5. 2026Beyond Semantics: Disentangling Information Scope in Sparse Autoencoders for CLIPcited by 1
  6. 2026From Tokens to Concepts: Leveraging SAE for SPLADEcited by 0
  7. 2026Latent Terms: Dense Retrievers Contain Trivially Extractable BM25-ready Zipfian Vocabulariescited by 0
  8. 2026Thinking vs. NoThinking: Towards Interpreting Reasoning Mechanisms of Large Language Models via Sparse Autoencoderscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Disentangling Dense Embeddings with Sparse Autoencoders
  2. 2026Aligning Sentence Embeddings to Human Concepts via Sparse Autoencoders
  3. 2026Xetrieval: Mechanistically Explaining Dense Retrieval
  4. 2022LED: Lexicon-Enlightened Dense Retriever for Large-Scale Retrieval
  5. 2025Interpretable Embeddings with Sparse Autoencoders: A Data Analysis Toolkit
  6. 2025Sparse and Dense Retrievers Learn Better Together: Joint Sparse-Dense Optimization for Text-Image Retrieval
  7. 2025Temporal Sparse Autoencoders: Leveraging the Sequential Nature of Language for Interpretability
  8. 2026UEmbed: Unified Sparse and Dense Multimodal Embeddings