mlsanityData to 7 Oct 2026

Retrieval-Pretrained Transformer: Long-range Language Modeling with Self-retrieval

arXiv 2306.13421 ↗First posted 23 Jun 2023cs.CLcited by 20

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

Retrieval-augmented language models (LMs) have received much attention recently. However, typically the retriever is not trained jointly as a native component of the LM, but added post-hoc to an already-pretrained LM, which limits the ability of the LM and the retriever to adapt to one another. In this work, we propose the Retrieval-Pretrained Transformer (RPT), an architecture and training procedure for jointly training a retrieval-augmented LM from scratch and apply it to the task of modeling long texts. Given a recently generated text chunk in a long document, the LM computes query representations, which are then used to retrieve earlier chunks in the document, located potentially tens of thousands of tokens before. Information from retrieved chunks is fused into the LM representations to predict the next target chunk. We train the retriever component with a semantic objective, where the goal is to retrieve chunks that increase the probability of the next chunk, according to a reference LM. We evaluate RPT on four long-range language modeling tasks, spanning books, code, and mathematical writing, and demonstrate that RPT improves retrieval quality and subsequently perplexity across the board compared to strong baselines.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 56 of its 66 references that are in the map. First posted on arXiv 23 Jun 2023.

Builds on

56 of 66 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Surface-Based Retrieval Reduces Perplexity of Retrieval-Augmented Language Models25 May
  2. 2023Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study13 Apr
  3. 2023In-Context Retrieval-Augmented Language Models31 Jan
  4. 2023REPLUG: Retrieval-Augmented Black-Box Language Models30 Jan
  5. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Models5 Aug
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks22 May
  7. 2020REALM: Retrieval-Augmented Language Model Pre-Training10 Feb

Other references · 49

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2020Longformer: The Long-Document Transformercited by 3,362
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  11. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  12. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
37 more
  1. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  2. 2020Reformer: The Efficient Transformercited by 1,658
  3. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  4. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  5. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  6. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  7. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  8. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  9. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  10. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  11. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  12. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  13. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  14. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  15. 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 282
  16. 2016Using the Output Embedding to Improve Language Modelscited by 266
  17. 2022Long Range Language Modeling via Gated State Spacescited by 265
  18. 2020Distilling Knowledge from Reader to Retriever for Question Answeringcited by 226
  19. 2022Memorizing Transformerscited by 219
  20. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  21. 2022Improving Passage Retrieval with Zero-Shot Question Generationcited by 164
  22. 2021Simple Entity-Centric Questions Challenge Dense Retrieverscited by 140
  23. 2021End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answeringcited by 133
  24. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 127
  25. 2022Training Language Models with Memory Augmentationcited by 112
  26. 2022Block-Recurrent Transformerscited by 107
  27. 2021Do Long-Range Language Models Actually Use Long-Range Context?cited by 92
  28. 2021Adaptive Semiparametric Language Modelscited by 83
  29. 2020Shortformer: Better Language Modeling using Shorter Inputscited by 76
  30. 2022Efficient Long-Text Understanding with Short-Text Modelscited by 71
  31. 2021Learning to Retrieve Passages without Supervisioncited by 46
  32. 2022Retrieval as Attention: End-to-end Learning of Retrieval and Reading within a Single Transformercited by 45
  33. 2022QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphscited by 42
  34. 2021Not All Memories are Created Equal: Learning to Forget by Expiringcited by 32
  35. 2020Do Transformers Need Deep Long-Range Memorycited by 24
  36. 2022Simplifying and Understanding State Space Models with Diagonal Linear RNNscited by 23
  37. 2023kNN-Adapter: Efficient Domain Adaptation for Black-Box Language Modelscited by 13

Built on by

Cited by year

Citing papers per year: 2023: 7; 2024: 10; 2025: 3.

20 papers.

Sort:

By area:

  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  2. 2023A Comprehensive Overview of Large Language Modelscited by 639
  3. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 364
  4. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 211
  5. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  6. 2023Attention Sorting Combats Recency Bias In Long Context Language Modelscited by 89
  7. 2024Reliable, Adaptable, and Attributable Language Models with Retrievalcited by 73
  8. 2023Recursively Summarizing Enables Long-Term Dialogue Memory in Large Language Modelscited by 62
  9. 2023KCTS: Knowledge-Constrained Tree Search Decoding with Token-Level Hallucination Detectioncited by 43
  10. 2024UniMS-RAG: A Unified Multi-source Retrieval-Augmented Generation for Personalized Dialogue Systemscited by 38
  11. 2024In Search of Needles in a 11M Haystack: Recurrent Memory Finds What LLMs Misscited by 36
  12. 2024Accelerating Retrieval-Augmented Language Model Serving with Speculationcited by 20
8 more
  1. 2024Associative Recurrent Memory Transformercited by 15
  2. 2024Analyzing and Adapting Large Language Models for Few-Shot Multilingual NLU: Are We There Yet?cited by 11
  3. 2024Efficient Length-Generalizable Attention via Causal Retrieval for Long-Context Language Modelingcited by 8
  4. 2025Hardware-aligned Hierarchical Sparse Attention for Efficient Long-term Memory Accesscited by 7
  5. 2025CoCoLex: Confidence-guided Copy-based Decoding for Grounded Legal Text Generationcited by 6
  6. 2024Intermediate Distillation: Data-Efficient Distillation from Black-Box LLMs for Information Retrievalcited by 3
  7. 2025Every Token Counts: Generalizing 16M Ultra-Long Context in Large Language Modelscited by 3
  8. 2024Retrieval Augmented Spelling Correction for E-Commerce Applicationscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processing
  2. 2023BRENT: Bidirectional Retrieval Enhanced Norwegian Transformer
  3. 2026To Memorize or to Retrieve: Scaling the Interaction Between Pretraining and Retrieval
  4. 2026Latent Abstraction for Retrieval-Augmented Generation
  5. 2024ACER: Automatic Language Model Context Extension via Retrieval
  6. 2025Direct Retrieval-augmented Optimization: Synergizing Knowledge Selection and Language Models
  7. 2024Unraveling and Mitigating Retriever Inconsistencies in Retrieval-Augmented Large Language Models
  8. 2024MemLong: Memory-Augmented Retrieval for Long Text Modeling