# Decoding Dense Embeddings: Sparse Autoencoders for Interpreting and Discretizing Dense Retrieval

arXiv: https://arxiv.org/abs/2506.00041 · HTML: https://mlsanity.com/paper/2506.00041/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability.
- Builds on 21 of its 45 references that are in the map.
- First posted on arXiv 28 May 2025.

## Placed in

- Mechanistic interpretability / [SAE features and their applications](https://mlsanity.com/mechinterp/sae-features-and-their-applications/)

## Builds on (most cited first, 21 of 21)

- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Large Dual Encoders Are Generalizable Retrievers](https://mlsanity.com/paper/2112.07899/) (2021)
- [Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models](https://mlsanity.com/paper/2403.19647/) (2024)
- [Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders](https://mlsanity.com/paper/2407.14435/) (2024)
- [Overview of the TREC 2019 deep learning track](https://mlsanity.com/paper/2003.07820/) (2020)
- [SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking](https://mlsanity.com/paper/2107.05720/) (2021)
- [Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling](https://mlsanity.com/paper/2104.06967/) (2021)
- [Overview of the TREC 2020 deep learning track](https://mlsanity.com/paper/2102.07662/) (2021)
- [COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted List](https://mlsanity.com/paper/2104.07186/) (2021)
- [SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval](https://mlsanity.com/paper/2207.02578/) (2022)
- [Sparse Autoencoders Do Not Find Canonical Units of Analysis](https://mlsanity.com/paper/2502.04878/) (2025)
- [SPLADE-v3: New baselines for SPLADE](https://mlsanity.com/paper/2403.06789/) (2024)
- [A Few Brief Notes on DeepImpact, COIL, and a Conceptual Framework for Information Retrieval Techniques](https://mlsanity.com/paper/2106.14807/) (2021)
- [Learning Passage Impacts for Inverted Indexes](https://mlsanity.com/paper/2104.12016/) (2021)
- [BMRetriever: Tuning Large Language Models as Better Biomedical Text Retrievers](https://mlsanity.com/paper/2404.18443/) (2024)
- [CITADEL: Conditional Token Interaction via Dynamic Lexical Routing for Efficient and Effective Multi-Vector Retrieval](https://mlsanity.com/paper/2211.10411/) (2022)
- [Interpret and Control Dense Retrieval with Sparse Latent Features](https://mlsanity.com/paper/2411.00786/) (2024)
- [Towards Axiomatic Explanations for Neural Ranking Models](https://mlsanity.com/paper/2106.08019/) (2021)
- [Explain like I am BM25: Interpreting a Dense Model's Ranked-List with a Sparse Approximation](https://mlsanity.com/paper/2304.12631/) (2023)
- [Interpreting Dense Retrieval as Mixture of Topics](https://mlsanity.com/paper/2111.13957/) (2021)

## Built on by (most cited first, 8 of 8)

- [Learning Retrieval Models with Sparse Autoencoders](https://mlsanity.com/paper/2603.13277/) (2026)
- [Visual Words Meet BM25: Sparse Auto-Encoder Visual Word Scoring for Image Retrieval](https://mlsanity.com/paper/2603.05781/) (2026)
- [QueStER: Query Specification for Generative keyword-based Retrieval](https://mlsanity.com/paper/2511.05301/) (2025)
- [Revolutionizing Long-Term Memory in AI: New Horizons with High-Capacity and High-Speed Storage](https://mlsanity.com/paper/2602.16192/) (2026)
- [Beyond Semantics: Disentangling Information Scope in Sparse Autoencoders for CLIP](https://mlsanity.com/paper/2604.05724/) (2026)
- [From Tokens to Concepts: Leveraging SAE for SPLADE](https://mlsanity.com/paper/2604.21511/) (2026)
- [Latent Terms: Dense Retrievers Contain Trivially Extractable BM25-ready Zipfian Vocabularies](https://mlsanity.com/paper/2605.29384/) (2026)
- [Thinking vs. NoThinking: Towards Interpreting Reasoning Mechanisms of Large Language Models via Sparse Autoencoders](https://mlsanity.com/paper/2608.08168/) (2026)

## Abstract (arXiv)

Despite their strong performance, Dense Passage Retrieval (DPR) models suffer from a lack of interpretability. In this work, we propose a novel interpretability framework that leverages Sparse Autoencoders (SAEs) to decompose previously uninterpretable dense embeddings from DPR models into distinct, interpretable latent concepts. We generate natural language descriptions for each latent concept, enabling human interpretations of both the dense embeddings and the query-document similarity scores of DPR models. We further introduce Concept-Level Sparse Retrieval (CL-SR), a retrieval framework that directly utilizes the extracted latent concepts as indexing units. CL-SR effectively combines the semantic expressiveness of dense embeddings with the transparency and efficiency of sparse representations. We show that CL-SR achieves high index-space and computational efficiency while maintaining robust performance across vocabulary and semantic mismatches.