mlsanityData to 7 Oct 2026

Provable Data Scaling Law for Meta Learning via Complexity Minimization

arXiv 2606.02008 ↗First posted 1 Jun 2026stat.ML cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Pre-training has become a fundamental paradigm in modern machine learning, with one of its key empirical benefits being reduced downstream sample complexity as the scale of pre-training data increases. However, existing theoretical frameworks for pre-training do not fully explain this phenomenon. In this paper, we introduce complexity minimization, a novel meta-representation learning framework designed to enable theoretical analysis of this scaling behavior, which learns representations by evaluating the downstream model complexity best suited to each domain and minimizing the worst-case such complexity across source domains. Our end-to-end theoretical analysis, spanning pre-training through downstream regression, shows that this framework provably captures this scaling behavior; in particular, we show that the error rate of few-shot adaptation improves as the amount of meta-training data grows. Empirically, we demonstrate that incorporating complexity regularization into existing meta-learning methods consistently improves downstream sample efficiency.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 65 of its 91 references that are in the map. First posted on arXiv 1 Jun 2026.

Builds on

65 of 91 references.

References · 65

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2023Segment Anythingcited by 8,158
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  7. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  8. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  9. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  10. 2016Matching Networks for One Shot Learningcited by 2,246
  11. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  12. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
38 more
  1. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  2. 2017Learning to Compare: Relation Network for Few-Shot Learningcited by 1,139
  3. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  4. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  5. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  6. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  7. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  8. 2018On Lazy Training in Differentiable Programmingcited by 671
  9. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  10. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  11. 2016Learning to learn by gradient descent by gradient descentcited by 616
  12. 2016Error bounds for approximations with deep ReLU networkscited by 567
  13. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  14. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  15. 2019Meta-Learning with Implicit Gradientscited by 523
  16. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  17. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  18. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 431
  19. 2018Meta-learning with differentiable closed-form solverscited by 395
  20. 2017Meta-SGD: Learning to Learn Quickly for Few-Shot Learningcited by 377
  21. 2021Explaining Neural Scaling Lawscited by 353
  22. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  23. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 309
  24. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  25. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  26. 2021Scaling Laws for Transfercited by 265
  27. 2019Uniform convergence may be unable to explain generalization in deep learningcited by 223
  28. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 220
  29. 2015The Benefit of Multitask Representation Learningcited by 194
  30. 2020Provable Meta-Learning of Linear Representationscited by 161
  31. 2018Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionalitycited by 160
  32. 2023Diffusion Models are Minimax Optimal Distribution Estimatorscited by 159
  33. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
  34. 2021Equinox: neural networks in JAX via callable PyTrees and filtered transformationscited by 143
  35. 2017Meta-Learning by Adjusting Priors Based on Extended PAC-Bayes Theorycited by 76
  36. 2019Nonparametric Regression on Low-Dimensional Manifolds using Deep ReLU Networks : Function Approximation and Statistical Recoverycited by 73
  37. 2013A PAC-Bayesian bound for Lifelong Learningcited by 69
  38. 2024How Feature Learning Can Improve Neural Scaling Lawscited by 55

50 most cited of 65 All 65 (JSON)

Built on by

Not cited yet.