mlsanityData to 14 Oct 2026

What Should Data Teach? Moving Bottlenecks Across Circuit, Store, and Use

arXiv 2609.32991 ↗First posted 26 Sep 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

What should data teach a language model at a particular point in training? A circuit view reveals three distinct bottlenecks: forming a computation, making its required content available, and selecting among available routes. A shared diagnosis-to-data principle connects them: localize the missing operation, preserve its causal relation, vary shortcut-bearing context, and re-audit the residual. Formation-sensitive selection and prerequisite ordering accelerate a binding-matching-transport path; a brief early prefix from the same training multiset retains a validation advantage through 100B tokens. Availability counterfactuals then distinguish writing content from invoking available memory, while paired supervision and context-opportunity ranking improve matched route decisions and long-context answer likelihood. A continuous 350M-model experiment connects the three interventions on the same facts: early circuit training improves subsequent learning, and the complete sequence outperforms stage-replacement controls on facts withheld from Use teaching. Independent query surfaces and opposed-source decisions expose conditional arbitration as the remaining frontier. Together, these results show why a change in the limiting operation calls for a change in supervision, not merely a new ranking of difficult examples.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 34 of its 34 references that are in the map. First posted on arXiv 26 Sep 2026.

Builds on

34 of 34 references.

References · 34

  1. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  3. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  4. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  5. 2022In-context Learning and Induction Headscited by 964
  6. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  7. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 458
  8. 2024Knowledge Conflicts for LLMs: A Surveycited by 283
  9. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 270
  10. 2024QuRating: Selecting High-Quality Data for Training Language Modelscited by 157
  11. 2018Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasiscited by 146
  12. 2023Physics of Language Models: Part 3.2, Knowledge Manipulationcited by 143
22 more
  1. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 121
  2. 2024MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Modelscited by 100
  3. 2024What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formationcited by 89
  4. 2024What is Wrong with Perplexity for Long-context Language Modeling?cited by 69
  5. 2024Capturing the Temporal Dependence of Training Data Influencecited by 35
  6. 2024Studying Large Language Model Behaviors Under Context-Memory Conflicts With Real Documentscited by 27
  7. 2025Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learningcited by 26
  8. 2025Preference Curriculum: LLMs Should Always Be Pretrained on Their Preferred Datacited by 16
  9. 2025Task Matters: Knowledge Requirements Shape LLM Responses to Context-Memory Conflictcited by 16
  10. 2025Group-Level Data Selection for Efficient Pretrainingcited by 13
  11. 2025LADM: Long-context Training Data Selection with Attention-based Dependency Measurement for LLMscited by 11
  12. 2026Mechanistic Data Attribution: Tracing the Training Origins of Interpretable LLM Unitscited by 10
  13. 2025LongAttn: Selecting Long-context Training Data via Token-level Attentioncited by 9
  14. 2026Why Supervised Fine-Tuning Fails to Learn: A Systematic Study of Incomplete Learning in Large Language Modelscited by 8
  15. 2025Influence Dynamics and Stagewise Data Attributioncited by 7
  16. 2025From Shortcut to Induction Head: How Data Diversity Shapes Algorithm Selection in Transformerscited by 6
  17. 2025Beyond Length: Quantifying Long-Range Information for Long-Context LLM Pretraining Datacited by 4
  18. 2025CASCADE Your Datasets for Cross-Mode Knowledge Retrieval of Language Modelscited by 2
  19. 2025Influence-driven Curriculum Learning for Pre-training on Limited Datacited by 2
  20. 2025Induction Signatures Are Not Enough: A Matched-Compute Study of Load-Bearing Structure in In-Context Learningcited by 2
  21. 2026Mechanistic Circuit Identification for Controllable Data Generationcited by 2
  22. 2026Symbolic Mechanistic Data Attribution: Tracing Training Influence to Learned Behavioral Policiescited by 1

Built on by

Not cited yet.