mlsanityData to 14 Oct 2026

Small-to-Large Generalization: Data Influences Models Consistently Across Scale

arXiv 2505.16260 ↗First posted 22 May 2025cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Choice of training data distribution greatly influences model behavior. Yet, in large-scale settings, precisely characterizing how changes in training data affects predictions is often difficult due to model training costs. Current practice is to instead extrapolate from scaled down, inexpensive-to-train proxy models. However, changes in data do not influence smaller and larger models identically. Therefore, understanding how choice of data affects large-scale models raises the question: how does training data distribution influence model behavior across compute scale? We find that small- and large-scale language model predictions (generally) do highly correlate across choice of training data. Equipped with these findings, we characterize how proxy scale affects effectiveness in two downstream proxy model applications: data attribution and dataset selection.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 57 of its 94 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

57 of 94 references.

References · 57

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  10. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  12. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
38 more
  1. 2022Emergent Abilities of Large Language Modelscited by 2,512
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  3. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  4. 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 629
  5. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
  6. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  7. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 523
  8. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 497
  9. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 496
  10. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 460
  11. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 399
  12. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
  13. 2022The Stack: 3 TB of permissively licensed source codecited by 359
  14. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  15. 2023Data Selection for Language Models via Importance Resamplingcited by 329
  16. 2023Studying Large Language Model Generalization with Influence Functionscited by 326
  17. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  18. 2023TRAK: Attributing Model Behavior at Scalecited by 283
  19. 2019Towards Efficient Data Valuation Based on the Shapley Valuecited by 281
  20. 2020Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depthcited by 245
  21. 2015Convergent Learning: Do different neural networks learn the same representations?cited by 220
  22. 2022Datamodels: Predicting Predictions from Training Datacited by 189
  23. 2018Representer Point Selection for Explaining Deep Neural Networkscited by 187
  24. 2022Training Data Influence Analysis and Estimation: A Surveycited by 135
  25. 2021Scaling Up Influence Functionscited by 132
  26. 2020What shapes feature representations? Exploring datasets, architectures, and trainingcited by 126
  27. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 121
  28. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 121
  29. 2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106
  30. 2024DsDm: Model-Aware Dataset Selection with Datamodelscited by 105
  31. 2024MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Modelscited by 100
  32. 2023Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networkscited by 97
  33. 2024What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functionscited by 94
  34. 2022More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalizecited by 91
  35. 2021Input Similarity from the Neural Network Perspectivecited by 79
  36. 2023Intriguing Properties of Data Attribution on Diffusion Modelscited by 47
  37. 2023The Journey, Not the Destination: How Data Guides Diffusion Modelscited by 43
  38. 2023Feature-Learning Networks Are Consistent Across Widths At Realistic Scalescited by 41

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Large Language Model Sourcing: A Surveycited by 5