# Cite Pretrain: Retrieval-Free Knowledge Attribution for Large Language Models

arXiv: https://arxiv.org/abs/2506.17585 · HTML: https://mlsanity.com/paper/2506.17585/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 63 of its 72 references that are in the map.
- First posted on arXiv 21 Jun 2025.

## Builds on (most cited first, 30 of 63)

- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [Enabling Large Language Models to Generate Text with Citations](https://mlsanity.com/paper/2305.14627/) (2023)
- [ELI5: Long Form Question Answering](https://mlsanity.com/paper/1907.09190/) (2019)
- [KILT: a Benchmark for Knowledge Intensive Language Tasks](https://mlsanity.com/paper/2009.02252/) (2020)
- [CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data](https://mlsanity.com/paper/1911.00359/) (2019)
- [Scaling Data-Constrained Language Models](https://mlsanity.com/paper/2305.16264/) (2023)
- [Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts](https://mlsanity.com/paper/2305.13300/) (2023)
- [FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation](https://mlsanity.com/paper/2310.03214/) (2023)
- [Evaluating Verifiability in Generative Search Engines](https://mlsanity.com/paper/2304.09848/) (2023)
- [Teaching language models to support answers with verified quotes](https://mlsanity.com/paper/2203.11147/) (2022)
- [RARR: Researching and Revising What Language Models Say, Using Language Models](https://mlsanity.com/paper/2210.08726/) (2022)
- [TRAK: Attributing Model Behavior at Scale](https://mlsanity.com/paper/2303.14186/) (2023)
- [Measuring Attribution in Natural Language Generation Models](https://mlsanity.com/paper/2112.12870/) (2021)
- [Transformer Memory as a Differentiable Search Index](https://mlsanity.com/paper/2202.06991/) (2022)
- [ASQA: Factoid Questions Meet Long-Form Answers](https://mlsanity.com/paper/2204.06092/) (2022)
- [Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity](https://mlsanity.com/paper/2310.07521/) (2023)
- [How Context Affects Language Models' Factual Predictions](https://mlsanity.com/paper/2005.04611/) (2020)
- [Rethinking with Retrieval: Faithful Large Language Model Inference](https://mlsanity.com/paper/2301.00303/) (2022)
- [Physics of Language Models: Part 3.2, Knowledge Manipulation](https://mlsanity.com/paper/2309.14402/) (2023)
- [Do Language Models Know When They're Hallucinating References?](https://mlsanity.com/paper/2305.18248/) (2023)
- [A Neural Corpus Indexer for Document Retrieval](https://mlsanity.com/paper/2206.02743/) (2022)
- [ExpertQA: Expert-Curated Questions and Attributed Answers](https://mlsanity.com/paper/2309.07852/) (2023)
- [From Matching to Generation: A Survey on Generative Information Retrieval](https://mlsanity.com/paper/2404.14851/) (2024)
- [Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Data](https://mlsanity.com/paper/2407.14985/) (2024)
- [TrueTeacher: Learning Factual Consistency Evaluation with Large Language Models](https://mlsanity.com/paper/2305.11171/) (2023)
- [LongCite: Enabling LLMs to Generate Fine-grained Citations in Long-context QA](https://mlsanity.com/paper/2409.02897/) (2024)

## Built on by (most cited first, 2 of 2)

- [Prototype Language Models](https://mlsanity.com/paper/2607.00510/) (2026)
- [Train, Retrieve, or Both? A Four-Arm Head-to-Head for Correct Statutory Citation on the Ontario Residential Tenancies Act](https://mlsanity.com/paper/2606.20359/) (2026)

## Abstract (arXiv)

Large language models can answer questions correctly and still hallucinate their citations. Most citation systems address this by retrieving documents at inference time. Retrieval adds latency and depends on external infrastructure. Retrieval noise can also affect the answer. We study whether LLMs can cite documents from their (continual) pretraining without test-time retrieval. We introduce CitePretrainBench to evaluate this setting. It includes Wikipedia, Common Crawl, and arXiv as real-world corpora and also includes novel, unseen documents. The benchmark includes short-form citation tasks for a single fact and long-form tasks for multiple facts. Continual-pretraining first binds factual knowledge to a persistent identifier for each document. Instruction tuning then teaches the model to put these identifiers in its answers as citations. Passive Indexing appends an identifier at the end of each document. Synthetic data used in Active Indexing restate facts in diverse, compositional forms. We train two mappings together. These mappings are source→fact and fact→source. The first direction involves the model generating content given a cited source. In the second, it learns to attribute its own answers when facts are paraphrased or combined. On Qwen-2.5-7B and 3B, Active Indexing improves citation precision over Passive Indexing on every evaluated citation task, with gains of up to 30.2 percentage points. In the scaling ablation, citation performance continues to improve as we add augmented data, including at 16 times the original token count. Combining internal and external citations also improves robustness to retrieval noise.