# Leveraging Decoder Architectures for Learned Sparse Retrieval

arXiv: https://arxiv.org/abs/2504.18151 · HTML: https://mlsanity.com/paper/2504.18151/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Apr 2025.

## Built on by (most cited first, 4 of 4)

- [Learning Retrieval Models with Sparse Autoencoders](https://mlsanity.com/paper/2603.13277/) (2026)
- [LACONIC: Dense-Level Effectiveness for Scalable Sparse Retrieval via a Two-Phase Training Curriculum](https://mlsanity.com/paper/2601.01684/) (2026)
- [On the Challenges and Opportunities of Learned Sparse Retrieval for Code](https://mlsanity.com/paper/2603.22008/) (2026)
- [To Case or Not to Case: An Empirical Study in Learned Sparse Retrieval](https://mlsanity.com/paper/2601.17500/) (2026)

## Abstract (arXiv)

Learned Sparse Retrieval (LSR) has traditionally focused on small-scale encoder-only transformer architectures. With the advent of large-scale pre-trained language models, their capability to generate sparse representations for retrieval tasks across different transformer-based architectures, including encoder-only, decoder-only, and encoder-decoder models, remains largely unexplored. This study investigates the effectiveness of LSR across these architectures, exploring various sparse representation heads and model scales. Our results highlight the limitations of using large language models to create effective sparse representations in zero-shot settings, identifying challenges such as inappropriate term expansions and reduced performance due to the lack of expansion. We find that the encoder-decoder architecture with multi-tokens decoding approach achieves the best performance among the three backbones. While the decoder-only model performs worse than the encoder-only model, it demonstrates the potential to outperform when scaled to a high number of parameters.