mlsanityData to 7 Oct 2026

Optimal Representation Size: High-Dimensional Analysis of Pretraining and Linear Probing

arXiv 2605.20105 ↗First posted 19 May 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Learning to generalise from limited data is a fundamental challenge for both artificial and biological systems. A common strategy is to extract reusable structure from abundant unlabelled data, enabling efficient adaptation to new tasks from limited labelled data. This two-stage paradigm is now standard in modern training pipelines, where pretraining is followed by fine-tuning or linear probing. We provide an analytical model of this process: structure extraction is formalized as principal component analysis on unlabelled data, and downstream learning as linear regression on a separate labelled dataset. In the high-dimensional regime, we derive exact expressions for training and generalisation error showcasing their dependence on representation dimensionality, unlabelled and labelled sample sizes, and task alignment. Our results show that pretrained representations strongly influence downstream generalisation, and we characterize the optimal representation size as a function of task parameters: with abundant pretraining data but scarce downstream data, maximally compressed representations are optimal, whereas with limited pretraining data, higher-dimensional representations generalise better. Furthermore, we establish an exact trade-off between pretraining and supervision, quantifying how much unlabelled data is required to replace a single labelled sample. Beyond our idealised model, we observe similar phenomenology in autoencoders and pretrained LLMs. Altogether, we highlight that optimising representation size is critical, giving conditions for when compression during pretraining improves generalisation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 53 of its 100 references that are in the map. First posted on arXiv 19 May 2026.

Builds on

53 of 100 references.

References · 53

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  10. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  11. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  12. 2019A Comprehensive Survey on Transfer Learningcited by 1,154
38 more
  1. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  2. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  3. 2019Benign Overfitting in Linear Regressioncited by 544
  4. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  5. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  6. 2017High-dimensional dynamics of generalization error in neural networkscited by 238
  7. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
  8. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  9. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 146
  10. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
  11. 2023The geometry of hidden representations of large transformer modelscited by 108
  12. 2020On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regressioncited by 97
  13. 2024The Surprising Effectiveness of Test-Time Training for Few-Shot Learningcited by 88
  14. 2018An analytic theory of generalization dynamics and transfer learning in deep linear networkscited by 84
  15. 2020Asymptotics of Ridge (less) Regression under General Source Conditioncited by 61
  16. 2019Overparameterized Neural Networks Implement Associative Memorycited by 45
  17. 2022Datasheet for the Pilecited by 41
  18. 2018From Principal Subspaces to Principal Components with Linear Autoencoderscited by 38
  19. 2024Asymptotics of feature learning in two-layer networks after one gradient-stepcited by 35
  20. 2023Gradient-Based Feature Learning under Structured Datacited by 32
  21. 2022The dynamics of representation learning in shallow, non-linear autoencoderscited by 27
  22. 2020Precise High-Dimensional Asymptotics for Quantifying Heterogeneous Transferscited by 26
  23. 2024Scaling laws for learning with real and surrogate datacited by 26
  24. 2023Towards a statistical theory of data selection under weak supervisioncited by 25
  25. 2021Probing transfer learning with a model of synthetic correlated datasetscited by 24
  26. 2024High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Lawscited by 21
  27. 2021On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependenciescited by 18
  28. 2024A Random Matrix Theory Perspective on the Spectrum of Learned Features and Asymptotic Generalization Capabilitiescited by 14
  29. 2024Generalization error of min-norm interpolators in transfer learningcited by 12
  30. 2024Features are fate: a theory of transfer learning in high-dimensional regressioncited by 10
  31. 2020Associative Memory in Iterated Overparameterized Sigmoid Autoencoderscited by 9
  32. 2025Spurious Correlations in High Dimensional Regression: The Roles of Regularization, Simplicity Bias and Over-Parameterizationcited by 9
  33. 2025Navigating the Latent Space Dynamics of Neural Modelscited by 7
  34. 2025Provable Benefits of Unsupervised Pre-training and Transfer Learning via Single-Index Modelscited by 6
  35. 2025High-dimensional Analysis of Synthetic Data Selectioncited by 5
  36. 2026A Theory of How Pretraining Shapes Inductive Bias in Fine-Tuningcited by 5
  37. 2017Effective learning is accompanied by high dimensional and efficient representations of neural activitycited by 4
  38. 2025Low Rank Gradients and Where to Find Themcited by 4

50 most cited of 53 All 53 (JSON)

Built on by

Not cited yet.