# Source-Aware Training Enables Knowledge Attribution in Language Models

arXiv: https://arxiv.org/abs/2404.01019 · HTML: https://mlsanity.com/paper/2404.01019/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 56 references that are in the map.
- First posted on arXiv 1 Apr 2024.

## Builds on (most cited first, 30 of 47)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Atlas: Few-shot Learning with Retrieval Augmented Language Models](https://mlsanity.com/paper/2208.03299/) (2022)
- [Show Your Work: Scratchpads for Intermediate Computation with Language Models](https://mlsanity.com/paper/2112.00114/) (2021)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [Enabling Large Language Models to Generate Text with Citations](https://mlsanity.com/paper/2305.14627/) (2023)
- [Towards Robust Interpretability with Self-Explaining Neural Networks](https://mlsanity.com/paper/1806.07538/) (2018)
- [The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"](https://mlsanity.com/paper/2309.12288/) (2023)
- [Detecting Pretraining Data from Large Language Models](https://mlsanity.com/paper/2310.16789/) (2023)
- [Autoregressive Entity Retrieval](https://mlsanity.com/paper/2010.00904/) (2020)
- [Studying Large Language Model Generalization with Influence Functions](https://mlsanity.com/paper/2308.03296/) (2023)
- [Evaluating Verifiability in Generative Search Engines](https://mlsanity.com/paper/2304.09848/) (2023)
- [Teaching language models to support answers with verified quotes](https://mlsanity.com/paper/2203.11147/) (2022)
- [RARR: Researching and Revising What Language Models Say, Using Language Models](https://mlsanity.com/paper/2210.08726/) (2022)
- [Physics of Language Models: Part 3.1, Knowledge Storage and Extraction](https://mlsanity.com/paper/2309.14316/) (2023)
- [Measuring Attribution in Natural Language Generation Models](https://mlsanity.com/paper/2112.12870/) (2021)
- [Transformer Memory as a Differentiable Search Index](https://mlsanity.com/paper/2202.06991/) (2022)
- [How Context Affects Language Models' Factual Predictions](https://mlsanity.com/paper/2005.04611/) (2020)
- [Large Language Models with Controllable Working Memory](https://mlsanity.com/paper/2211.05110/) (2022)

## Built on by (most cited first, 30 of 30)

- [From Matching to Generation: A Survey on Generative Information Retrieval](https://mlsanity.com/paper/2404.14851/) (2024)
- [Trustworthiness in Retrieval-Augmented Generation Systems: A Survey](https://mlsanity.com/paper/2409.10102/) (2024)
- [CiteME: Can Language Models Accurately Cite Scientific Claims?](https://mlsanity.com/paper/2407.12861/) (2024)
- [Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded Attributions and Learning to Refuse](https://mlsanity.com/paper/2409.11242/) (2024)
- [ALR2: A Retrieve-then-Reason Framework for Long-context Question Answering](https://mlsanity.com/paper/2410.03227/) (2024)
- [OG-RAG: Ontology-Grounded Retrieval-Augmented Generation For Large Language Models](https://mlsanity.com/paper/2412.15235/) (2024)
- [Metadata Conditioning Accelerates Language Model Pre-training](https://mlsanity.com/paper/2501.01956/) (2025)
- [Verifiable Generation with Subsentence-Level Fine-Grained Citations](https://mlsanity.com/paper/2406.06125/) (2024)
- [Abstention vs. Hallucination: Benchmarking LLM Source Attribution for Scientific Citations](https://mlsanity.com/paper/2405.02228/) (2024)
- [Combinatorial Optimization for All: Using LLMs to Aid Non-Experts in Improving Optimization Algorithms](https://mlsanity.com/paper/2503.10968/) (2025)
- [Attribution, Citation, and Quotation: A Survey of Evidence-based Text Generation with Large Language Models](https://mlsanity.com/paper/2508.15396/) (2025)
- [Large Language Model Sourcing: A Survey](https://mlsanity.com/paper/2510.10161/) (2025)
- [Influential Training Data Retrieval for Explaining Verbalized Confidence of LLMs](https://mlsanity.com/paper/2601.10645/) (2026)
- [When Does Metadata Conditioning (NOT) Work for Language Model Pre-Training? A Study with Context-Free Grammars](https://mlsanity.com/paper/2504.17562/) (2025)
- [The Attribution Crisis in LLM Search Results](https://mlsanity.com/paper/2508.00838/) (2025)
- [Enhancing elusive clues in knowledge learning by contrasting attention of language models](https://mlsanity.com/paper/2409.17954/) (2024)
- [Can Performant LLMs Be Ethical? Quantifying the Impact of Web Crawling Opt-Outs](https://mlsanity.com/paper/2504.06219/) (2025)
- [Cite Pretrain: Retrieval-Free Knowledge Attribution for Large Language Models](https://mlsanity.com/paper/2506.17585/) (2025)
- [Reflections and New Directions for Human-Centered Large Language Models](https://mlsanity.com/paper/2605.06901/) (2026)
- [Evaluation of Attribution Bias in Generator-Aware Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2410.12380/) (2024)
- [Probing for Knowledge Attribution in Large Language Models](https://mlsanity.com/paper/2602.22787/) (2026)
- [RUBEN: Rule-Based Explanations for Retrieval-Augmented LLM Systems](https://mlsanity.com/paper/2605.10862/) (2026)
- [CAGE: Cognitive Attribution Graphs for Faithful Inline Citation Generation in Long-Form Question Answering](https://mlsanity.com/paper/2607.24236/) (2026)
- [The SMeL Test: A simple benchmark for media literacy in language models](https://mlsanity.com/paper/2508.02074/) (2025)
- [Rule-Based Explanations for Retrieval-Augmented LLM Systems](https://mlsanity.com/paper/2510.22689/) (2025)
- [Beyond URLs: Metadata Diversity and Position for Efficient LLM Pretraining](https://mlsanity.com/paper/2511.21613/) (2025)
- [LIME: Making LLM Data More Efficient with Linguistic Metadata Embeddings](https://mlsanity.com/paper/2512.07522/) (2025)
- [Source-Modality Monitoring in Vision-Language Models](https://mlsanity.com/paper/2604.22038/) (2026)
- [Chain of Evidence: Pixel-Level Visual Attribution for Iterative Retrieval-Augmented Generation](https://mlsanity.com/paper/2605.01284/) (2026)
- [DataDignity: Training Data Attribution for Large Language Models](https://mlsanity.com/paper/2605.05687/) (2026)

## Abstract (arXiv)

Large language models (LLMs) learn a vast amount of knowledge during pretraining, but they are often oblivious to the source(s) of such knowledge. We investigate the problem of intrinsic source citation, where LLMs are required to cite the pretraining source supporting a generated response. Intrinsic source citation can enhance LLM transparency, interpretability, and verifiability. To give LLMs such ability, we explore source-aware training -- a recipe that involves (i) training the LLM to associate unique source document identifiers with the knowledge in each document, followed by (ii) an instruction-tuning stage to teach the LLM to cite a supporting pretraining source when prompted. Source-aware training borrows from existing pretraining/fine-tuning frameworks and requires minimal changes to the model architecture or implementation. Through experiments on synthetic data, we demonstrate that our training recipe can enable faithful attribution to the pretraining data without a substantial impact on the model's perplexity compared to standard pretraining. Our findings also highlight the importance of pretraining data augmentation in achieving attribution. Code and data available here: urlhttps://github.com/mukhal/intrinsic-source-citation