mlsanityData to 7 Oct 2026

Pre-computed memory or on-the-fly encoding? A hybrid approach to retrieval augmentation makes the most of your compute

arXiv 2301.10448 ↗First posted 25 Jan 2023cs.CL cs.AI cs.LGcited by 17

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

Retrieval-augmented language models such as Fusion-in-Decoder are powerful, setting the state of the art on a variety of knowledge-intensive tasks. However, they are also expensive, due to the need to encode a large number of retrieved passages. Some work avoids this cost by pre-encoding a text corpus into a memory and retrieving dense representations directly. However, pre-encoding memory incurs a severe quality penalty as the memory representations are not conditioned on the current input. We propose LUMEN, a hybrid between these two extremes, pre-computing the majority of the retrieval representation and completing the encoding on the fly using a live encoder that is conditioned on the question and fine-tuned for the task. We show that LUMEN significantly outperforms pure memory on multiple question-answering tasks while being much cheaper than FiD, and outperforms both for any given compute budget. Moreover, the advantage of LUMEN over FiD increases with model size.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 36 of its 49 references that are in the map. First posted on arXiv 25 Jan 2023.

Builds on

36 of 49 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022FiDO: Fusion-in-Decoder optimized for stronger performance and faster inference15 Dec
  2. 2022An Efficient Memory-Augmented Transformer for Knowledge-Intensive NLP Tasks30 Oct
  3. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Models5 Aug
  4. 2022Augmenting Pre-trained Language Models with QA-Memory for Open-Domain Question Answering10 Apr
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks22 May
  6. 2020REALM: Retrieval-Augmented Language Model Pre-Training10 Feb

Other references · 30

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  4. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  5. 2020Knowledge Distillation: A Surveycited by 1,578
  6. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  7. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  8. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  9. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  10. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  11. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  12. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
18 more
  1. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  2. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  3. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  4. 2022Efficiently Scaling Transformer Inferencecited by 529
  5. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  6. 2022Generate rather than Retrieve: Large Language Models are Strong Context Generatorscited by 296
  7. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  8. 2022Memorizing Transformerscited by 219
  9. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  10. 2022Training Language Models with Memory Augmentationcited by 112
  11. 2021KG-FiD: Infusing Knowledge Graph in Fusion-in-Decoder for Open-Domain Question Answeringcited by 66
  12. 2022FiD-Light: Efficient and Effective Retrieval-Augmented Text Generationcited by 63
  13. 2021Mention Memory: incorporating textual knowledge into Transformers through entity mention attentioncited by 42
  14. 2021Rider: Reader-Guided Passage Reranking for Open-Domain Question Answeringcited by 28
  15. 2022Decoupled Context Processing for Context Augmented Language Modelingcited by 24
  16. 2018Adaptive Document Retrieval for Deep Question Answeringcited by 16
  17. 2021ReadTwice: Reading Very Large Documents with Memoriescited by 14
  18. 2022Can Open-Domain QA Reader Utilize External Knowledge Efficiently like Humans?cited by 10

Built on by

Cited by year

Citing papers per year: 2023: 6; 2024: 6; 2025: 5.

17 papers.

Sort:

By area:

  1. 2025A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Modelscited by 85
  2. 2024Retrieval-Augmented Generation for Natural Language Processing: A Surveycited by 70
  3. 2024Bridging the Preference Gap between Retrievers and LLMscited by 69
  4. 2025A Survey on Knowledge-Oriented Retrieval-Augmented Generationcited by 34
  5. 2023Retrieval-augmented Multi-modal Chain-of-Thoughts Reasoning for Large Language Modelscited by 18
  6. 2023ATLANTIC: Structure-Aware Retrieval-Augmented Language Model for Interdisciplinary Sciencecited by 15
  7. 2024Retrieval-Enhanced Machine Learning: Synthesis and Opportunitiescited by 11
  8. 2023LAIT: Efficient Multi-Segment Encoding in Transformers with Layer-Adjustable Interactioncited by 7
  9. 2023GLIMMER: generalized late-interaction memory rerankercited by 7
  10. 2023BTR: Binary Token Representations for Efficient Retrieval Augmented Language Modelscited by 5
  11. 2024RAG Foundry: A Framework for Enhancing LLMs for Retrieval Augmented Generationcited by 5
  12. 2024Attendre: Wait To Attend By Retrieval With Evicted Queries in Memory-Based Transformers for Long Context Processingcited by 3
5 more
  1. 2024Recall: Empowering Multimodal Embedding for Edge Devicescited by 3
  2. 2025NAACL2025 Tutorial: Adaptation of Large Language Modelscited by 2
  3. 2023MEMORY-VQ: Compression for Tractable Internet-Scale Memorycited by 1
  4. 2025DACL-RAG: Data Augmentation Strategy with Curriculum Learning for Retrieval-Augmented Generationcited by 1
  5. 2025DeCAL Tokenwise Compressioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026To Memorize or to Retrieve: Scaling the Interaction Between Pretraining and Retrieval
  2. 2025MLP Memory: A Retriever-Pretrained Memory for Large Language Models
  3. 2023Retrieval-Pretrained Transformer: Long-range Language Modeling with Self-retrieval
  4. 2024MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentation
  5. 2023BRENT: Bidirectional Retrieval Enhanced Norwegian Transformer
  6. 2026EnComp: Lightweight Encoder-Only Context Compression for Retrieval-Augmented Question Answering
  7. 2023RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation
  8. 2024Memory3: Language Modeling with Explicit Memory