# Optimizing Retrieval-augmented Reader Models via Token Elimination

arXiv: https://arxiv.org/abs/2310.13682 · HTML: https://mlsanity.com/paper/2310.13682/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation.
- Builds on 27 of its 39 references that are in the map.
- First posted on arXiv 20 Oct 2023.

## Placed in

- Retrieval-augmented generation / [RAG robustness and adaptive retrieval](https://mlsanity.com/rag/rag-robustness-and-adaptive-retrieval/)

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [Atlas: Few-shot Learning with Retrieval Augmented Language Models](https://mlsanity.com/paper/2208.03299/) (2022)
- [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [ELI5: Long Form Question Answering](https://mlsanity.com/paper/1907.09190/) (2019)
- [KILT: a Benchmark for Knowledge Intensive Language Tasks](https://mlsanity.com/paper/2009.02252/) (2020)
- [Depth-Adaptive Transformer](https://mlsanity.com/paper/1910.10073/) (2019)
- [Distilling Knowledge from Reader to Retriever for Question Answering](https://mlsanity.com/paper/2012.04584/) (2020)
- [Hurdles to Progress in Long-form Question Answering](https://mlsanity.com/paper/2103.06332/) (2021)
- [The Right Tool for the Job: Matching Model and Instance Complexities](https://mlsanity.com/paper/2004.07453/) (2020)
- [Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models](https://mlsanity.com/paper/2210.03162/) (2022)
- [CoLT5: Faster Long-Range Transformers with Conditional Computation](https://mlsanity.com/paper/2303.09752/) (2023)
- [Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search](https://mlsanity.com/paper/2010.07003/) (2020)
- [FiD-Light: Efficient and Effective Retrieval-Augmented Text Generation](https://mlsanity.com/paper/2209.14290/) (2022)
- [Read before Generate! Faithful Long Form Question Answering with Machine Reading](https://mlsanity.com/paper/2203.00343/) (2022)
- [FiDO: Fusion-in-Decoder optimized for stronger performance and faster inference](https://mlsanity.com/paper/2212.08153/) (2022)
- [Nugget: Neural Agglomerative Embeddings of Text](https://mlsanity.com/paper/2310.01732/) (2023)
- [GLIMMER: generalized late-interaction memory reranker](https://mlsanity.com/paper/2306.10231/) (2023)

## Built on by (most cited first, 10 of 10)

- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [From Persona to Personalization: A Survey on Role-Playing Language Agents](https://mlsanity.com/paper/2404.18231/) (2024)
- [Transformers are Multi-State RNNs](https://mlsanity.com/paper/2401.06104/) (2024)
- [A Survey on Retrieval-Augmented Text Generation for Large Language Models](https://mlsanity.com/paper/2404.10981/) (2024)
- [Selective Attention Improves Transformer](https://mlsanity.com/paper/2410.02703/) (2024)
- [SirLLM: Streaming Infinite Retentive LLM](https://mlsanity.com/paper/2405.12528/) (2024)
- [Rethinking Token Reduction for State Space Models](https://mlsanity.com/paper/2410.14725/) (2024)
- [MORE: Multi-mOdal REtrieval Augmented Generative Commonsense Reasoning](https://mlsanity.com/paper/2402.13625/) (2024)
- [Preference-Guided Refactored Tuning for Retrieval Augmented Code Generation](https://mlsanity.com/paper/2409.15895/) (2024)
- [Attendre: Wait To Attend By Retrieval With Evicted Queries in Memory-Based Transformers for Long Context Processing](https://mlsanity.com/paper/2401.04881/) (2024)

## Abstract (arXiv)

Fusion-in-Decoder (FiD) is an effective retrieval-augmented language model applied across a variety of open-domain tasks, such as question answering, fact checking, etc. In FiD, supporting passages are first retrieved and then processed using a generative model (Reader), which can cause a significant bottleneck in decoding time, particularly with long outputs. In this work, we analyze the contribution and necessity of all the retrieved passages to the performance of reader models, and propose eliminating some of the retrieved information, at the token level, that might not contribute essential information to the answer generation process. We demonstrate that our method can reduce run-time by up to 62.2%, with only a 2% reduction in performance, and in some cases, even improve the performance results.