# OLMoTrace: Tracing Language Model Outputs Back to Trillions of Training Tokens

arXiv: https://arxiv.org/abs/2504.07096 · HTML: https://mlsanity.com/paper/2504.07096/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 11 references that are in the map.
- First posted on arXiv 9 Apr 2025.

## Builds on (most cited first, 7 of 7)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [OLMoE: Open Mixture-of-Experts Language Models](https://mlsanity.com/paper/2409.02060/) (2024)
- [Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens](https://mlsanity.com/paper/2401.17377/) (2024)
- [Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions](https://mlsanity.com/paper/2005.06676/) (2020)
- [Training Language Models to Generate Text with Citations via Fine-grained Rewards](https://mlsanity.com/paper/2402.04315/) (2024)
- [ORCA: Interpreting Prompted Language Models via Locating Supporting Data Evidence in the Ocean of Pretraining Data](https://mlsanity.com/paper/2205.12600/) (2022)

## Built on by (most cited first, 27 of 27)

- [Infini-gram mini: Exact n-gram Search at the Internet Scale with FM-Index](https://mlsanity.com/paper/2506.12229/) (2025)
- [QuCo-RAG: Quantifying Uncertainty from the Pre-training Corpus for Dynamic Retrieval-Augmented Generation](https://mlsanity.com/paper/2512.19134/) (2025)
- [Gemma Needs Help: Investigating and Mitigating Emotional Instability in LLMs](https://mlsanity.com/paper/2603.10011/) (2026)
- [Measuring LLM Novelty As The Frontier Of Original And High-Quality Output](https://mlsanity.com/paper/2504.09389/) (2025)
- ["Lost-in-the-Later": Framework for Quantifying Contextual Grounding in Large Language Models](https://mlsanity.com/paper/2507.05424/) (2025)
- [DATE-LM: Benchmarking Data Attribution Evaluation for Large Language Models](https://mlsanity.com/paper/2507.09424/) (2025)
- [Overstating Attitudes, Ignoring Networks: LLM Biases in Simulating Misinformation Susceptibility](https://mlsanity.com/paper/2602.04674/) (2026)
- [One Model, Many Morals: Uncovering Cross-Linguistic Misalignments in Computational Moral Reasoning](https://mlsanity.com/paper/2509.21443/) (2025)
- [Influential Training Data Retrieval for Explaining Verbalized Confidence of LLMs](https://mlsanity.com/paper/2601.10645/) (2026)
- [ExPLAIND: Unifying Model, Data, and Training Attribution to Study Model Behavior](https://mlsanity.com/paper/2505.20076/) (2025)
- [Tracing the Data Trail: A Survey of Data Provenance, Transparency and Traceability in LLMs](https://mlsanity.com/paper/2601.14311/) (2026)
- [Cite Pretrain: Retrieval-Free Knowledge Attribution for Large Language Models](https://mlsanity.com/paper/2506.17585/) (2025)
- [Revisiting the Past: Data Unlearning with Model State History](https://mlsanity.com/paper/2506.20941/) (2025)
- [NanoKnow: How to Know What Your Language Model Knows](https://mlsanity.com/paper/2602.20122/) (2026)
- [How Open Must Language Models be to Enable Reliable Scientific Inference?](https://mlsanity.com/paper/2603.26539/) (2026)
- [Generative AI floods and dilutes the market for books](https://mlsanity.com/paper/2607.20349/) (2026)
- [A Bayesian Information-Theoretic Approach to Data Attribution](https://mlsanity.com/paper/2604.03858/) (2026)
- [Why are all LLMs Obsessed with Japanese Culture? On the Hidden Cultural and Regional Biases of LLMs](https://mlsanity.com/paper/2604.21751/) (2026)
- [SuperValid: Capability-Aligned OOD Validation for Generalizable Downstream Scaling](https://mlsanity.com/paper/2605.28179/) (2026)
- [LLMs Can Leak Training Data But Do They Want To? A Propensity-Aware Evaluation of Memorization in LLMs](https://mlsanity.com/paper/2606.06286/) (2026)
- [LLM generation novelty through the lens of semantic similarity](https://mlsanity.com/paper/2510.27313/) (2025)
- [TrackList: Tracing Back Query Linguistic Diversity for Head and Tail Knowledge in Open Large Language Models](https://mlsanity.com/paper/2511.21006/) (2025)
- [Feature Starvation as Geometric Instability in Sparse Autoencoders](https://mlsanity.com/paper/2605.05341/) (2026)
- [Efficient and Scalable Provenance Tracking for LLM-Generated Code Snippets](https://mlsanity.com/paper/2605.28510/) (2026)
- [Quantifying the Agreement Between Data-Influence and Data-Similarity to Understand LLM Behavior](https://mlsanity.com/paper/2606.23591/) (2026)
- [Tracing LLM Behavior to the Training Data with Empirical Next-Token Distributions](https://mlsanity.com/paper/2607.14306/) (2026)
- [What to Forget in Unlearning? Forget Set Curation for Language Models](https://mlsanity.com/paper/2608.14855/) (2026)

## Abstract (arXiv)

We present OLMoTrace, the first system that traces the outputs of language models back to their full, multi-trillion-token training data in real time. OLMoTrace finds and shows verbatim matches between segments of language model output and documents in the training text corpora. Powered by an extended version of infini-gram (Liu et al., 2024), our system returns tracing results within a few seconds. OLMoTrace can help users understand the behavior of language models through the lens of their training data. We showcase how it can be used to explore fact checking, hallucination, and the creativity of language models. OLMoTrace is publicly available and fully open-source.