mlsanityData to 7 Oct 2026

Optimizing Retrieval-augmented Reader Models via Token Elimination

arXiv 2310.13682 ↗First posted 20 Oct 2023cs.CL cs.AI cs.LGcited by 10

On the map: Retrieval-augmented generation → RAG robustness and adaptive retrieval

Abstract (arXiv)

Fusion-in-Decoder (FiD) is an effective retrieval-augmented language model applied across a variety of open-domain tasks, such as question answering, fact checking, etc. In FiD, supporting passages are first retrieved and then processed using a generative model (Reader), which can cause a significant bottleneck in decoding time, particularly with long outputs. In this work, we analyze the contribution and necessity of all the retrieved passages to the performance of reader models, and propose eliminating some of the retrieved information, at the token level, that might not contribute essential information to the answer generation process. We demonstrate that our method can reduce run-time by up to 62.2%, with only a 2% reduction in performance, and in some cases, even improve the performance results.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation. Builds on 27 of its 39 references that are in the map. First posted on arXiv 20 Oct 2023.

Builds on

27 of 39 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  7. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  8. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  9. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  10. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  11. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  12. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
15 more
  1. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  2. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  3. 2019ELI5: Long Form Question Answeringcited by 567
  4. 2019Depth-Adaptive Transformercited by 227
  5. 2020Distilling Knowledge from Reader to Retriever for Question Answeringcited by 226
  6. 2021Hurdles to Progress in Long-form Question Answeringcited by 166
  7. 2020The Right Tool for the Job: Matching Model and Instance Complexitiescited by 125
  8. 2022Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Modelscited by 96
  9. 2023CoLT5: Faster Long-Range Transformers with Conditional Computationcited by 84
  10. 2020Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Searchcited by 76
  11. 2022FiD-Light: Efficient and Effective Retrieval-Augmented Text Generationcited by 63
  12. 2022Read before Generate! Faithful Long Form Question Answering with Machine Readingcited by 47
  13. 2022FiDO: Fusion-in-Decoder optimized for stronger performance and faster inferencecited by 31
  14. 2023Nugget: Neural Agglomerative Embeddings of Textcited by 19
  15. 2023GLIMMER: generalized late-interaction memory rerankercited by 7

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 9.

10 papers.

  1. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  2. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 206
  3. 2024Transformers are Multi-State RNNscited by 120
  4. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Modelscited by 77
  5. 2024Selective Attention Improves Transformercited by 26
  6. 2024SirLLM: Streaming Infinite Retentive LLMcited by 16
  7. 2024Rethinking Token Reduction for State Space Modelscited by 14
  8. 2024MORE: Multi-mOdal REtrieval Augmented Generative Commonsense Reasoningcited by 12
  9. 2024Preference-Guided Refactored Tuning for Retrieval Augmented Code Generationcited by 10
  10. 2024Attendre: Wait To Attend By Retrieval With Evicted Queries in Memory-Based Transformers for Long Context Processingcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2026ReFilter: Improving Robustness of Retrieval-Augmented Generation via Gated Filter
  2. 2024Context Quality Matters in Training Fusion-in-Decoder for Extractive Open-Domain Question Answering
  3. 2025LFD: Layer Fused Decoding to Exploit External Knowledge in Retrieval-Augmented Generation
  4. 2025Parallel Key-Value Cache Fusion for Position Invariant RAG
  5. 2023Modeling Uncertainty and Using Post-fusion as Fallback Improves Retrieval Augmented Generation with LLMs
  6. 2025Retrieval-Augmented Generation: A Comprehensive Survey of Architectures, Enhancements, and Robustness Frontiers
  7. 2026Guaranteeing Knowledge Integration with Joint Decoding for Retrieval-Augmented Generation
  8. 2025From Retrieval to Generation: Comparing Different Approaches