# Parameter-Efficient Sparse Retrievers and Rerankers using Adapters

arXiv: https://arxiv.org/abs/2303.13220 · HTML: https://mlsanity.com/paper/2303.13220/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 32 references that are in the map.
- First posted on arXiv 23 Mar 2023.

## Builds on (most cited first, 30 of 30)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks](https://mlsanity.com/paper/2106.04489/) (2021)
- [Pretrained Transformers for Text Ranking: BERT and Beyond](https://mlsanity.com/paper/2010.06467/) (2020)
- [AdapterDrop: On the Efficiency of Adapters in Transformers](https://mlsanity.com/paper/2010.11918/) (2020)
- [Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models](https://mlsanity.com/paper/2203.06904/) (2022)
- [Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2004.03829/) (2020)
- [From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective](https://mlsanity.com/paper/2205.04733/) (2022)
- [Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation](https://mlsanity.com/paper/2104.08771/) (2021)
- [Minimizing FLOPs to Learn Efficient Sparse Representations](https://mlsanity.com/paper/2004.05665/) (2020)
- [Robust Transfer Learning with Pretrained Language Models through Adapters](https://mlsanity.com/paper/2108.02340/) (2021)
- [An Efficiency Study for SPLADE Models](https://mlsanity.com/paper/2207.03834/) (2022)
- [TREC Deep Learning Track: Reusable Test Collections in the Large Data Regime](https://mlsanity.com/paper/2104.09399/) (2021)
- [TripClick: The Log Files of a Large Health Web Search Engine](https://mlsanity.com/paper/2103.07901/) (2021)
- [Parameter-Efficient Prompt Tuning Makes Generalized and Calibrated Neural Text Retrievers](https://mlsanity.com/paper/2207.07087/) (2022)
- [Parameter-Efficient Neural Reranking for Cross-Lingual and Multilingual Retrieval](https://mlsanity.com/paper/2204.02292/) (2022)
- [Parameter-Efficient Abstractive Question Answering over Tables or Text](https://mlsanity.com/paper/2204.03357/) (2022)
- [AdapterHub Playground: Simple and Flexible Few-Shot Learning with Adapters](https://mlsanity.com/paper/2108.08103/) (2021)
- [Semi-Siamese Bi-encoder Neural Ranking Model Using Lightweight Fine-Tuning](https://mlsanity.com/paper/2110.14943/) (2021)
- [Sparsifying Sparse Representations for Passage Retrieval by Top-k Masking](https://mlsanity.com/paper/2112.09628/) (2021)
- [Scattered or Connected? An Optimized Parameter-efficient Tuning Approach for Information Retrieval](https://mlsanity.com/paper/2208.09847/) (2022)
- [Establishing Strong Baselines for TripClick Health Retrieval](https://mlsanity.com/paper/2201.00365/) (2022)

## Built on by (most cited first, 3 of 3)

- [Query-driven Relevant Paragraph Extraction from Legal Judgments](https://mlsanity.com/paper/2404.00595/) (2024)
- [Q-PEFT: Query-dependent Parameter Efficient Fine-tuning for Text Reranking with Large Language Models](https://mlsanity.com/paper/2404.04522/) (2024)
- [Back to Basics: A Simple Recipe for Improving Out-of-Domain Retrieval in Dense Encoders](https://mlsanity.com/paper/2311.09765/) (2023)

## Abstract (arXiv)

Parameter-Efficient transfer learning with Adapters have been studied in Natural Language Processing (NLP) as an alternative to full fine-tuning. Adapters are memory-efficient and scale well with downstream tasks by training small bottle-neck layers added between transformer layers while keeping the large pretrained language model (PLMs) frozen. In spite of showing promising results in NLP, these methods are under-explored in Information Retrieval. While previous studies have only experimented with dense retriever or in a cross lingual retrieval scenario, in this paper we aim to complete the picture on the use of adapters in IR. First, we study adapters for SPLADE, a sparse retriever, for which adapters not only retain the efficiency and effectiveness otherwise achieved by finetuning, but are memory-efficient and orders of magnitude lighter to train. We observe that Adapters-SPLADE not only optimizes just 2% of training parameters, but outperforms fully fine-tuned counterpart and existing parameter-efficient dense IR models on IR benchmark datasets. Secondly, we address domain adaptation of neural retrieval thanks to adapters on cross-domain BEIR datasets and TripClick. Finally, we also consider knowledge sharing between rerankers and first stage rankers. Overall, our study complete the examination of adapters for neural IR