# Large Language Models are Built-in Autoregressive Search Engines

arXiv: https://arxiv.org/abs/2305.09612 · HTML: https://mlsanity.com/paper/2305.09612/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 36 references that are in the map.
- First posted on arXiv 16 May 2023.

## Builds on (most cited first, 28 of 28)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval](https://mlsanity.com/paper/2007.00808/) (2020)
- [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017)
- [KILT: a Benchmark for Knowledge Intensive Language Tasks](https://mlsanity.com/paper/2009.02252/) (2020)
- [RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2010.08191/) (2020)
- [Autoregressive Entity Retrieval](https://mlsanity.com/paper/2010.00904/) (2020)
- [Generate rather than Retrieve: Large Language Models are Strong Context Generators](https://mlsanity.com/paper/2209.10063/) (2022)
- [Transformer Memory as a Differentiable Search Index](https://mlsanity.com/paper/2202.06991/) (2022)
- [Promptagator: Few-shot Dense Retrieval From 8 Examples](https://mlsanity.com/paper/2209.11755/) (2022)
- [Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering](https://mlsanity.com/paper/2101.00774/) (2021)
- [End-to-End Open-Domain Question Answering with BERTserini](https://mlsanity.com/paper/1902.01718/) (2019)
- [Dense Text Retrieval based on Pretrained Language Models: A Survey](https://mlsanity.com/paper/2211.14876/) (2022)
- [A Survey of Knowledge-Enhanced Text Generation](https://mlsanity.com/paper/2010.04389/) (2020)
- [Autoregressive Search Engines: Generating Substrings as Document Identifiers](https://mlsanity.com/paper/2204.10628/) (2022)
- [A Neural Corpus Indexer for Document Retrieval](https://mlsanity.com/paper/2206.02743/) (2022)
- [CREAK: A Dataset for Commonsense Reasoning over Entity Knowledge](https://mlsanity.com/paper/2109.01653/) (2021)
- [Relevance-guided Supervision for OpenQA with ColBERT](https://mlsanity.com/paper/2007.00814/) (2020)
- [KG-FiD: Infusing Knowledge Graph in Fusion-in-Decoder for Open-Domain Question Answering](https://mlsanity.com/paper/2110.04330/) (2021)
- [A Survey of Multi-task Learning in Natural Language Processing: Regarding Task Relatedness and Training Methods](https://mlsanity.com/paper/2204.03508/) (2022)
- [Rider: Reader-Guided Passage Reranking for Open-Domain Question Answering](https://mlsanity.com/paper/2101.00294/) (2021)
- [Grape: Knowledge Graph Enhanced Passage Reader for Open-domain Question Answering](https://mlsanity.com/paper/2210.02933/) (2022)
- [Knowledge-Aware Procedural Text Understanding with Multi-Stage Training](https://mlsanity.com/paper/2009.13199/) (2020)
- [A Unified Encoder-Decoder Framework with Entity Memory](https://mlsanity.com/paper/2210.03273/) (2022)

## Built on by (most cited first, 27 of 27)

- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023)
- [Large Language Models for Information Retrieval: A Survey](https://mlsanity.com/paper/2308.07107/) (2023)
- [Language to Rewards for Robotic Skill Synthesis](https://mlsanity.com/paper/2306.08647/) (2023)
- [Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization](https://mlsanity.com/paper/2406.01171/) (2024)
- [LawBench: Benchmarking Legal Knowledge of Large Language Models](https://mlsanity.com/paper/2309.16289/) (2023)
- [From Matching to Generation: A Survey on Generative Information Retrieval](https://mlsanity.com/paper/2404.14851/) (2024)
- [How Does Generative Retrieval Scale to Millions of Passages?](https://mlsanity.com/paper/2305.11841/) (2023)
- [Semantic Gesticulator: Semantics-Aware Co-Speech Gesture Synthesis](https://mlsanity.com/paper/2405.09814/) (2024)
- [How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advances](https://mlsanity.com/paper/2310.07343/) (2023)
- [IfQA: A Dataset for Open-domain Question Answering under Counterfactual Presuppositions](https://mlsanity.com/paper/2305.14010/) (2023)
- [A Survey of Generative Search and Recommendation in the Era of Large Language Models](https://mlsanity.com/paper/2404.16924/) (2024)
- [Hire Your Anthropologist! Rethinking Culture Benchmarks Through an Anthropological Lens](https://mlsanity.com/paper/2510.05931/) (2025)
- [Retrieval-Augmented Visual Question Answering via Built-in Autoregressive Search Engines](https://mlsanity.com/paper/2502.16641/) (2025)
- [Dense Passage Retrieval: Is it Retrieving?](https://mlsanity.com/paper/2402.11035/) (2024)
- [Beyond Relevance: Evaluate and Improve Retrievers on Perspective Awareness](https://mlsanity.com/paper/2405.02714/) (2024)
- [Evaluating Generative Ad Hoc Information Retrieval](https://mlsanity.com/paper/2311.04694/) (2023)
- [Chaos with Keywords: Exposing Large Language Models Sycophantic Hallucination to Misleading Keywords and Evaluating Defense Strategies](https://mlsanity.com/paper/2406.03827/) (2024)
- [Lightweight and Direct Document Relevance Optimization for Generative Information Retrieval](https://mlsanity.com/paper/2504.05181/) (2025)
- [Positional Biases Shift as Inputs Approach Context Window Limits](https://mlsanity.com/paper/2508.07479/) (2025)
- [Beyond Memorization: The Challenge of Random Memory Access in Language Models](https://mlsanity.com/paper/2403.07805/) (2024)
- [Explaining Tree Model Decisions in Natural Language for Network Intrusion Detection](https://mlsanity.com/paper/2310.19658/) (2023)
- [On Synthetic Data Strategies for Domain-Specific Generative Retrieval](https://mlsanity.com/paper/2502.17957/) (2025)
- [Towards a Middleware for Large Language Models](https://mlsanity.com/paper/2411.14513/) (2024)
- [Evaluating Chinese Large Language Models: The Influence of Persona Assignment on Stereotypes and Safeguards](https://mlsanity.com/paper/2506.04975/) (2025)
- [OKG: On-the-Fly Keyword Generation in Sponsored Search Advertising](https://mlsanity.com/paper/2412.03577/) (2024)
- [Context-Aware Scientific Knowledge Extraction on Linked Open Data using Large Language Models](https://mlsanity.com/paper/2506.17580/) (2025)
- [DiverValue-Bench: A Benchmark and Fine-Tuning Framework for Aligning Large Language Models with Diverse Human Values](https://mlsanity.com/paper/2509.08022/) (2025)

## Abstract (arXiv)

Document retrieval is a key stage of standard Web search engines. Existing dual-encoder dense retrievers obtain representations for questions and documents independently, allowing for only shallow interactions between them. To overcome this limitation, recent autoregressive search engines replace the dual-encoder architecture by directly generating identifiers for relevant documents in the candidate pool. However, the training cost of such autoregressive search engines rises sharply as the number of candidate documents increases. In this paper, we find that large language models (LLMs) can follow human instructions to directly generate URLs for document retrieval. Surprisingly, when providing a few Query-URL pairs as in-context demonstrations, LLMs can generate Web URLs where nearly 90% of the corresponding documents contain correct answers to open-domain questions. In this way, LLMs can be thought of as built-in search engines, since they have not been explicitly trained to map questions to document identifiers. Experiments demonstrate that our method can consistently achieve better retrieval performance than existing retrieval approaches by a significant margin on three open-domain question answering benchmarks, under both zero and few-shot settings. The code for this work can be found at urlhttps://github.com/Ziems/llm-url.