mlsanityData to 14 Oct 2026

Shift is Good: Mismatched Data Mixing Improves Test Performance

arXiv 2510.25108 ↗First posted 29 Oct 2025cs.LG stat.MLcited by 5

Not on the map. Why?

Abstract (arXiv)

We consider training and testing on mixture distributions with different training and test proportions. We show that in many settings, and in some sense generically, distribution shift can be beneficial, and test performance can improve due to mismatched training proportions, even if the components are unrelated and with no transfer between components. In a variety of scenarios, we identify the optimal training proportions and the extent to which such distribution shift can be beneficial. We show how the same analysis applies also to a compositional setting with differing distribution of component "skills'' at training and test.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 17 of its 24 references that are in the map. First posted on arXiv 29 Oct 2025.

Builds on

17 of 24 references.

References · 17

  1. 2020Scaling Laws for Neural Language Modelscited by 6,586
  2. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  3. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
  4. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 399
  5. 2023Data Selection for Language Models via Importance Resamplingcited by 329
  6. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 184
  7. 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 177
  8. 2023Efficient Online Data Mixing For Language Model Pre-Trainingcited by 77
  9. 2025Scaling Laws for Optimal Data Mixturescited by 52
  10. 2024Adaptive Data Optimization: Dynamic Sample Selection with Scaling Lawscited by 47
  11. 2016Efficient Distributed Learning with Sparsitycited by 38
  12. 2024Data Selection via Optimal Control for Language Modelscited by 38
5 more
  1. 2025Learning Compositional Functions with Transformers from Easy-to-Hard Datacited by 36
  2. 2023Provable Advantage of Curriculum Learning on Parity Targets with Mixed Inputscited by 22
  3. 2021Out-of-Distribution Generalization in Kernel Regressioncited by 21
  4. 2015Distributed Multitask Learningcited by 11
  5. 2023Metalearning with Very Few Samples Per Taskcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Outcome-Based RL Provably Leads Transformers to Reason, but Only With the Right Datacited by 6
  2. 2026Positive Distribution Shift as a Framework for Understanding Tractable Learningcited by 3
  3. 2026Explaining Data Mixing Scaling Lawscited by 1
  4. 2026Learning through Internalizationcited by 1
  5. 2026The Power of Power Law: Asymmetry Enables Compositional Reasoningcited by 0