mlsanityData to 7 Oct 2026

Autoregressive Search Engines: Generating Substrings as Document Identifiers

arXiv 2204.10628 ↗First posted 22 Apr 2022cs.CL cs.IRcited by 112

Not on the map. Why?

Abstract (arXiv)

Knowledge-intensive language tasks require NLP systems to both provide the correct answer and retrieve supporting evidence for it in a given corpus. Autoregressive language models are emerging as the de-facto standard for generating answers, with newer and more powerful systems emerging at an astonishing pace. In this paper we argue that all this (and future) progress can be directly applied to the retrieval problem with minimal intervention to the models' architecture. Previous work has explored ways to partition the search space into hierarchical structures and retrieve documents by autoregressively generating their unique identifier. In this work we propose an alternative that doesn't force any structure in the search space: using all ngrams in a passage as its possible identifiers. This setup allows us to use an autoregressive model to generate and score distinctive ngrams, that are then mapped to full passages through an efficient data structure. Empirically, we show this not only outperforms prior autoregressive approaches but also leads to an average improvement of at least 10 points over more established retrieval solutions for passage-level retrieval on the KILT benchmark, establishing new state-of-the-art downstream performance on some datasets, while using a considerably lighter memory footprint than competing systems. Code and pre-trained models at https://github.com/facebookresearch/SEAL.

Key facts. Cited by 112 papers in the map (data through 7 Oct 2026). Builds on 55 of its 77 references that are in the map. First posted on arXiv 22 Apr 2022.

Builds on

55 of 77 references.

References · 55

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  10. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  11. 2019Language Models as Knowledge Bases?cited by 1,922
  12. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
38 more
  1. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  2. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  3. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  4. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  5. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  6. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  7. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  8. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  9. 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 425
  10. 2020Autoregressive Entity Retrievalcited by 333
  11. 2021SPLADE: Sparse Lexical and Expansion Model for First Stage Rankingcited by 283
  12. 2022Transformer Memory as a Differentiable Search Indexcited by 242
  13. 2019Document Expansion by Query Predictioncited by 225
  14. 2021Internet-Augmented Dialogue Generationcited by 217
  15. 2021PAQ: 65 Million Probably-Asked Questions and What You Can Do With Themcited by 198
  16. 2020Generation-Augmented Retrieval for Open-domain Question Answeringcited by 186
  17. 2020How Context Affects Language Models' Factual Predictionscited by 181
  18. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  19. 2022Improving Passage Retrieval with Zero-Shot Question Generationcited by 164
  20. 2020Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasetscited by 137
  21. 2022Internet-augmented language models through few-shot prompting for open-domain question answeringcited by 131
  22. 2021SPLADE v2: Sparse Lexical and Expansion Model for Information Retrievalcited by 126
  23. 2021COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted Listcited by 113
  24. 2022Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completioncited by 102
  25. 2020Learning Dense Representations of Phrases at Scalecited by 91
  26. 2021A Few Brief Notes on DeepImpact, COIL, and a Conceptual Framework for Information Retrieval Techniquescited by 69
  27. 2021Learning Passage Impacts for Inverted Indexescited by 68
  28. 2019Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrievalcited by 65
  29. 2021The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpuscited by 60
  30. 2021Efficient Passage Retrieval with Hashing for Open-domain Question Answeringcited by 53
  31. 2021Domain-matched Pre-training Tasks for Dense Retrievalcited by 53
  32. 2020SPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrievalcited by 50
  33. 2021Multilingual Autoregressive Entity Linkingcited by 50
  34. 2020SparTerm: Learning Term-based Sparse Representation for Fast Text Retrievalcited by 45
  35. 2021Multi-task Retrieval for Knowledge-Intensive Taskscited by 44
  36. 2021Hindsight: Posterior-guided training of retrievers for improved open-ended generationcited by 38
  37. 2021Phrase Retrieval Learns Passage Retrieval, Toocited by 33
  38. 2019How Decoding Strategies Affect the Verifiability of Generated Textcited by 27

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 14; 2023: 27; 2024: 36; 2025: 19; 2026: 16.

112 papers.

Sort:

By area:

  1. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  2. 2022Precise Zero-Shot Dense Retrieval without Relevance Labelscited by 508
  3. 2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Modelscited by 414
  4. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  5. 2022Generate rather than Retrieve: Large Language Models are Strong Context Generatorscited by 296
  6. 2022Re2G: Retrieve, Rerank, Generatecited by 102
  7. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  8. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Modelscited by 77
  9. 2024VISTA: Visualized Text Embedding For Universal Multi-Modal Retrievalcited by 67
  10. 2022Recitation-Augmented Language Modelscited by 65
  11. 2022FiD-Light: Efficient and Effective Retrieval-Augmented Text Generationcited by 63
  12. 2025Chain-of-Retrieval Augmented Generationcited by 48
38 more
  1. 2022Bridging the Gap Between Indexing and Retrieval for Differentiable Search Index with Query Generationcited by 47
  2. 2023How Does Generative Retrieval Scale to Millions of Passages?cited by 47
  3. 2023Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized Language Modelscited by 46
  4. 2023Learning to Rank in Generative Retrievalcited by 46
  5. 2022CorpusBrain: Pre-train a Generative Retrieval Model for Knowledge-Intensive Language Taskscited by 36
  6. 2024PEDANTS: Cheap but Effective and Interpretable Answer Equivalencecited by 31
  7. 2024Stochastic RAG: End-to-End Retrieval-Augmented Generation through Expected Utility Maximizationcited by 29
  8. 2022What Are You Token About? Dense Retrieval as Distributions Over the Vocabularycited by 28
  9. 2023Trends in Integration of Knowledge and Large Language Models: A Survey and Taxonomy of Methods, Benchmarks, and Applicationscited by 28
  10. 2024Generative Cross-Modal Retrieval: Memorizing Images in Multimodal Language Models for Retrieval and Beyondcited by 27
  11. 2023Large Language Models are Built-in Autoregressive Search Enginescited by 26
  12. 2024ALR2: A Retrieve-then-Reason Framework for Long-context Question Answeringcited by 25
  13. 2024Evaluating n-Gram Novelty of Language Models Using Rusty-DAWGcited by 24
  14. 2024RetroLLM: Empowering Large Language Models to Retrieve Fine-grained Evidence within Generationcited by 20
  15. 2024Planning Ahead in Generative Retrieval: Guiding Autoregressive Generation through Simultaneous Decodingcited by 19
  16. 2023IRGen: Generative Modeling for Image Retrievalcited by 18
  17. 2024CorpusLM: Towards a Unified Language Model on Corpus for Knowledge-Intensive Taskscited by 18
  18. 2023GLEN: Generative Retrieval via Lexical Index Learningcited by 17
  19. 2023Multiview Identifiers Enhanced Generative Retrievalcited by 16
  20. 2024Unifying Generative and Dense Retrieval for Sequential Recommendationcited by 16
  21. 2023Expand, Rerank, and Retrieve: Query Reranking for Open-Domain Question Answeringcited by 15
  22. 2023Context Aware Query Rewriting for Text Rankers using LLMcited by 14
  23. 2024A Survey of Generative Search and Recommendation in the Era of Large Language Modelscited by 14
  24. 2025DynamicRAG: Leveraging Outputs of Large Language Model as Feedback for Dynamic Reranking in Retrieval-Augmented Generationcited by 14
  25. 2025Retrieval-Augmented Visual Question Answering via Built-in Autoregressive Search Enginescited by 13
  26. 2025Generative Retrieval and Alignment Model: A New Paradigm for E-commerce Retrievalcited by 13
  27. 2025Infini-gram mini: Exact n-gram Search at the Internet Scale with FM-Indexcited by 13
  28. 2024Dense Passage Retrieval: Is it Retrieving?cited by 12
  29. 2023Synergistic Interplay between Search and Large Language Models for Information Retrievalcited by 11
  30. 2023GripRank: Bridging the Gap between Retrieval and Generation via the Generative Knowledge Improved Passage Rankingcited by 11
  31. 2023Evaluating Generative Ad Hoc Information Retrievalcited by 11
  32. 2024Generative Dense Retrieval: Memory Can Be a Burdencited by 11
  33. 2024MINERS: Multilingual Language Models as Semantic Retrieverscited by 11
  34. 2025Learning Facts at Scale with Active Readingcited by 11
  35. 2022Multi-Task Retrieval-Augmented Text Generation with Relevance Samplingcited by 10
  36. 2024CorpusBrain++: A Continual Generative Pre-Training Framework for Knowledge-Intensive Language Taskscited by 9
  37. 2024Generative Retrieval with Preference Optimization for E-commerce Searchcited by 9
  38. 2025Lightweight and Direct Document Relevance Optimization for Generative Information Retrievalcited by 9

50 most cited of 112 All 112 (JSON)