mlsanityData to 14 Oct 2026

Tracing LLM Behavior to the Training Data with Empirical Next-Token Distributions

arXiv 2607.14306 ↗First posted 15 Jul 2026cs.AIcited by 0

On the map: Mechanistic interpretability → SAE features and their applications

Abstract (arXiv)

In this paper, we study the connection between an LLM's output distribution and the data used to train it. Specifically, we study the degree to which an LLM's next-token distribution agrees with the empirical next-token distribution (ENTD) given the context in the training data. The ENTD is an appealing target because it is the unrestricted global minimizer of the next-token cross entropy loss used for pretraining, as well as an easily interpretable function of the pretraining corpus. We find that for a significant fraction of inputs, the LLM's distribution agrees with the ENTD almost perfectly, and the agreement generally increases with model scale and training compute. Nevertheless, there is a long tail of input sequences where the LLM and ENTD differ significantly, and we examine several possible sources of this discrepancy across the transformer architecture, training procedure, and finite-sample noise in the ENTD estimate itself. More broadly, we hope our findings will encourage more work on ``data-centric mechanistic interpretability,'' a complement to standard mechanistic interpretability that opens the black box of how model behaviors arise from the data, rather than how they are encoded in the learned weights.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability. Builds on 24 of its 31 references that are in the map. First posted on arXiv 15 Jul 2026.

Builds on

24 of 31 references.

References · 24

  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  3. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
  4. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  5. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 399
  6. 2023Data Selection for Language Models via Importance Resamplingcited by 329
  7. 2023Studying Large Language Model Generalization with Influence Functionscited by 326
  8. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 270
  9. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 184
  10. 2024Stealing Part of a Production Language Modelcited by 159
  11. 2024Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokenscited by 153
  12. 2017Breaking the Softmax Bottleneck: A High-Rank RNN Language Modelcited by 152
12 more
  1. 2025Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)cited by 141
  2. 2022Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Modelscited by 92
  3. 2024Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Datacited by 85
  4. 2021Frequency Effects on Syntactic Rule Learning in Transformerscited by 59
  5. 2022Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictionscited by 57
  6. 2025PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runscited by 35
  7. 2025OLMoTrace: Tracing Language Model Outputs Back to Trillions of Training Tokenscited by 27
  8. 2024Understanding Transformers via N-gram Statisticscited by 26
  9. 2026Mechanistic Data Attribution: Tracing the Training Origins of Interpretable LLM Unitscited by 10
  10. 2026Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retentioncited by 10
  11. 2025Quantitative Bounds for Length Generalization in Transformerscited by 8
  12. 2026How Do Transformers Learn to Associate Tokens: Gradient Leading Terms Bring Mechanistic Interpretabilitycited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025I Predict Therefore I Am: Is Next Token Prediction Enough to Learn Human-Interpretable Concepts from Data?
  2. 2026LUMOS: Tracing Parametric Knowledge from Training Data to Behavioral Outputs in LLMs
  3. 2024Beyond the Black Box: A Statistical Model for LLM Reasoning and Inference
  4. 2025Grokking in LLM Pretraining? Monitor Memorization-to-Generalization without Test
  5. 2026InfoSFT: Learn More and Forget Less with Information-Aware Token Weighting
  6. 2025On multi-token prediction for efficient LLM inference
  7. 2026Unlocking Lossless Speedups in LLMs via Discrete Diffusion
  8. 2025How Post-Training Reshapes LLMs: A Mechanistic View on Knowledge, Truthfulness, Refusal, and Confidence