mlsanityData to 7 Oct 2026

Decoupled Context Processing for Context Augmented Language Modeling

arXiv 2210.05758 ↗First posted 11 Oct 2022cs.CL cs.AI cs.IRcited by 24

Not on the map. Why?

Abstract (arXiv)

Language models can be augmented with a context retriever to incorporate knowledge from large external databases. By leveraging retrieved context, the neural network does not have to memorize the massive amount of world knowledge within its internal parameters, leading to better parameter efficiency, interpretability and modularity. In this paper we examined a simple yet effective architecture for incorporating external context into language models based on decoupled Encoder Decoder architecture. We showed that such a simple architecture achieves competitive results on auto-regressive language modeling and open domain question answering tasks. We also analyzed the behavior of the proposed model which performs grounded context transfer. Finally we discussed the computational implications of such retrieval augmented models.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 36 of its 57 references that are in the map. First posted on arXiv 11 Oct 2022.

Builds on

36 of 57 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  10. 2017Billion-scale similarity search with GPUscited by 2,477
  11. 2020Extracting Training Data from Large Language Modelscited by 2,144
  12. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
24 more
  1. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  2. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  3. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  4. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  5. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  6. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  7. 2017A Closer Look at Memorization in Deep Networkscited by 936
  8. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  9. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 853
  10. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  11. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  12. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  13. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  14. 2018Universal Sentence Encodercited by 485
  15. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  16. 2017Efficient Natural Language Response Suggestion for Smart Replycited by 228
  17. 2020Distilling Knowledge from Reader to Retriever for Question Answeringcited by 226
  18. 2021End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answeringcited by 133
  19. 2020Pre-training via Paraphrasingcited by 92
  20. 2020Learning Dense Representations of Phrases at Scalecited by 91
  21. 2021Adaptive Semiparametric Language Modelscited by 83
  22. 2019Real-Time Open-Domain Question Answering with Dense-Sparse Phrase Indexcited by 64
  23. 2021Mention Memory: incorporating textual knowledge into Transformers through entity mention attentioncited by 42
  24. 2021Attention Approximates Sparse Distributed Memorycited by 29

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 10; 2024: 7; 2025: 4; 2026: 1.

24 papers.

Sort:

By area:

  1. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  2. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
  3. 2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Modelscited by 414
  4. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  5. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 121
  6. 2025A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Modelscited by 85
  7. 2024Retrieval-Augmented Generation for Natural Language Processing: A Surveycited by 70
  8. 2022FiDO: Fusion-in-Decoder optimized for stronger performance and faster inferencecited by 31
  9. 2023Chameleon: a Heterogeneous and Disaggregated Accelerator System for Retrieval-Augmented Language Modelscited by 23
  10. 2023Pre-computed memory or on-the-fly encoding? A hybrid approach to retrieval augmentation makes the most of your computecited by 17
  11. 2023ResMem: Learn what you can and memorize the restcited by 13
  12. 2023Characterizing Attribution and Fluency Tradeoffs for Retrieval-Augmented Large Language Modelscited by 13
12 more
  1. 2024Retrieval-Enhanced Machine Learning: Synthesis and Opportunitiescited by 11
  2. 2023LAIT: Efficient Multi-Segment Encoding in Transformers with Layer-Adjustable Interactioncited by 7
  3. 2023GLIMMER: generalized late-interaction memory rerankercited by 7
  4. 2024ReFusion: Improving Natural Language Understanding with Computation-Efficient Retrieval Representation Fusioncited by 5
  5. 2024RAEE: A Robust Retrieval-Augmented Early Exit Framework for Efficient Inferencecited by 4
  6. 2025Synthetic bootstrapped pretrainingcited by 4
  7. 2024COBIAS: Assessing the Contextual Reliability of Bias Benchmarks for Language Modelscited by 3
  8. 2025Is Visual in-Context Learning for Compositional Medical Tasks within Reach?cited by 3
  9. 2025EmoRAG: Evaluating RAG Robustness to Symbolic Perturbationscited by 3
  10. 2024R-Eval: A Unified Toolkit for Evaluating Domain Knowledge of Retrieval Augmented Large Language Modelscited by 2
  11. 2023MEMORY-VQ: Compression for Tractable Internet-Scale Memorycited by 1
  12. 2026AMV-L: Lifecycle-Managed Agent Memory for Tail-Latency Control in Long-Running LLM Systemscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Improving Retrieval Augmented Open-Domain Question-Answering with Vectorized Contexts
  2. 2024Context Quality Matters in Training Fusion-in-Decoder for Extractive Open-Domain Question Answering
  3. 2023BRENT: Bidirectional Retrieval Enhanced Norwegian Transformer
  4. 2024Adaptive Contrastive Decoding in Retrieval-Augmented Generation for Handling Noisy Contexts
  5. 2024ACER: Automatic Language Model Context Extension via Retrieval
  6. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Context
  7. 2025Efficient Knowledge Feeding to Language Models: A Novel Integrated Encoder-Decoder Architecture
  8. 2023RAVEN: In-Context Learning with Retrieval-Augmented Encoder-Decoder Language Models