mlsanityData to 7 Oct 2026

Transformer Memory as a Differentiable Search Index

arXiv 2202.06991 ↗First posted 14 Feb 2022cs.CL cs.AI cs.IRcited by 242

Not on the map. Why?

Abstract (arXiv)

In this paper, we demonstrate that information retrieval can be accomplished with a single Transformer, in which all information about the corpus is encoded in the parameters of the model. To this end, we introduce the Differentiable Search Index (DSI), a new paradigm that learns a text-to-text model that maps string queries directly to relevant docids; in other words, a DSI model answers queries directly using only its parameters, dramatically simplifying the whole retrieval process. We study variations in how documents and their identifiers are represented, variations in training procedures, and the interplay between models and corpus sizes. Experiments demonstrate that given appropriate design choices, DSI significantly outperforms strong baselines such as dual encoder models. Moreover, DSI demonstrates strong generalization capabilities, outperforming a BM25 baseline in a zero-shot setup.

Key facts. Cited by 242 papers in the map (data through 7 Oct 2026). Builds on 32 of its 45 references that are in the map. First posted on arXiv 14 Feb 2022.

Builds on

32 of 45 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  8. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  9. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  10. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  11. 2019Language Models as Knowledge Bases?cited by 1,922
  12. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
20 more
  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  2. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  3. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  4. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  5. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  6. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  7. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  8. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  9. 2020Hopfield Networks is All You Needcited by 461
  10. 2020Autoregressive Entity Retrievalcited by 333
  11. 2019Document Expansion by Query Predictioncited by 225
  12. 2017The Case for Learned Index Structurescited by 188
  13. 2020Generation-Augmented Retrieval for Open-domain Question Answeringcited by 186
  14. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  15. 2021Scale Efficiently: Insights from Pre-training and Fine-tuning Transformerscited by 105
  16. 2021Exploring the Limits of Large Scale Pre-trainingcited by 104
  17. 2017Neural Ranking Models with Weak Supervisioncited by 61
  18. 2021GenIE: Generative Information Extractioncited by 53
  19. 2018End-to-End Retrieval in Continuous Spacecited by 46
  20. 2020Learned Indexes for a Google-scale Disk-based Databasecited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 27; 2023: 47; 2024: 60; 2025: 51; 2026: 57.

242 papers.

Sort:

By area:

  1. 2022Emergent Abilities of Large Language Modelscited by 2,483
  2. 2022Precise Zero-Shot Dense Retrieval without Relevance Labelscited by 508
  3. 2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Modelscited by 414
  4. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 356
  5. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  6. 2023Large Language Models are Effective Text Rankers with Pairwise Ranking Promptingcited by 302
  7. 2022Generate rather than Retrieve: Large Language Models are Strong Context Generatorscited by 296
  8. 2022UL2: Unifying Language Learning Paradigmscited by 292
  9. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 218
  10. 2023Recommender Systems with Generative Retrievalcited by 191
  11. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 158
  12. 2023Large Language Models as Zero-Shot Conversational Recommenderscited by 126
38 more
  1. 2022RankT5: Fine-Tuning T5 for Text Ranking with Ranking Lossescited by 114
  2. 2022Autoregressive Search Engines: Generating Substrings as Document Identifierscited by 112
  3. 2023LawBench: Benchmarking Legal Knowledge of Large Language Modelscited by 111
  4. 2022Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Modelscited by 108
  5. 2023ExpertQA: Expert-Curated Questions and Attributed Answerscited by 96
  6. 2023Practical Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibrationcited by 90
  7. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  8. 2023Crawling the Internal Knowledge-Base of Language Modelscited by 85
  9. 2024Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalizationcited by 78
  10. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  11. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Modelscited by 77
  12. 2023A Survey of Large Language Models Attributioncited by 71
  13. 2022Recitation-Augmented Language Modelscited by 65
  14. 2022FiD-Light: Efficient and Effective Retrieval-Augmented Text Generationcited by 63
  15. 2023Leveraging Large Language Models in Conversational Recommender Systemscited by 63
  16. 2022Transcending Scaling Laws with 0.1% Extra Computecited by 55
  17. 2024Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?cited by 53
  18. 2024DeTeCtive: Detecting AI-generated Text via Multi-Level Contrastive Learningcited by 52
  19. 2023OpenICL: An Open-Source Framework for In-context Learningcited by 51
  20. 2022Bridging the Gap Between Indexing and Retrieval for Differentiable Search Index with Query Generationcited by 47
  21. 2023How Does Generative Retrieval Scale to Millions of Passages?cited by 47
  22. 2022Retrieval-Enhanced Machine Learningcited by 46
  23. 2023Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized Language Modelscited by 46
  24. 2023Learning to Rank in Generative Retrievalcited by 46
  25. 2022QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphscited by 42
  26. 2023In-Context Learning with Many Demonstration Examplescited by 39
  27. 2022CorpusBrain: Pre-train a Generative Retrieval Model for Knowledge-Intensive Language Taskscited by 36
  28. 2022DSI++: Updating Transformer Memory with New Documentscited by 35
  29. 2023A Survey on Knowledge Editing of Neural Networkscited by 34
  30. 2025Query-Focused Retrieval Heads Improve Long-Context Reasoning and Re-rankingcited by 30
  31. 2023Information Retrieval Meets Large Language Models: A Strategic Report from Chinese IR Communitycited by 29
  32. 2024Source-Aware Training Enables Knowledge Attribution in Language Modelscited by 29
  33. 2022What Are You Token About? Dense Retrieval as Distributions Over the Vocabularycited by 28
  34. 2023NevIR: Negation in Neural Information Retrievalcited by 28
  35. 2023Trends in Integration of Knowledge and Large Language Models: A Survey and Taxonomy of Methods, Benchmarks, and Applicationscited by 28
  36. 2024C-RAG: Certified Generation Risks for Retrieval-Augmented Language Modelscited by 27
  37. 2024Generative Cross-Modal Retrieval: Memorizing Images in Multimodal Language Models for Retrieval and Beyondcited by 27
  38. 2025PLUM: Adapting Pre-trained Language Models for Industrial-scale Generative Recommendationscited by 27

50 most cited of 242 All 242 (JSON)