mlsanityData to 7 Oct 2026

The Geometry of Saturation: Effective Rank Predicts When Labels Stop Helping in Few-Shot Classification

arXiv 2606.24903 ↗First posted 12 Jun 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Few-shot label acquisition lacks a label-free signal for when additional labels cease to improve accuracy: existing stopping criteria either require a held-out validation set (violating the few-shot premise) or rely on theoretically ungrounded heuristics, so we introduce the spectral saturation index S(K)=erank(hatSigmaW(K))/K, the exponential spectral entropy of the pooled within-class covariance normalized by per-class support size K, which measures the exploration rate per label and falls below a fixed threshold τ=0.02 once the explored spectral subspace saturates and marginal accuracy gains vanish; across 49 real tasks (binary, 5-way, 10-way) and three frozen backbones (PCA-50, CLIP ViT-B/32, DINOv2 ViT-S/14), S(K) correlates strongly with the marginal gain on doubling the support set (ρpool=0.6366, p=2.9×10-57, cluster-bootstrap 95% CI [0.551,0.720]), a fixed τ=0.02 classifies stop/continue decisions with cluster-bootstrap AUC=0.787 (95% CI [0.713,0.860]) with high recall on meaningful gains (Delta A>1%), and a partial correlation controlling for log K yields ρpartial=0.324 (p=1.65×10-13), confirming S(K) carries spectral information beyond shared K-dependence; theory predicts this from first principles, since the population effective rank sets the saturation scale Ksat≈erank(SigmaW)/τ, τ=0.02 sits at the boundary between the first and second descent (Nakkiran et al., 2021), and O(1/K) bias in the sample effective rank explains the small-K hump in S(K); for unregularized linear probes (C=∞), practitioners should halt when S(K)<0.02 (PCA-50, hard stop) or monitor S(K) dropping from ~0.3→0.05 (foundation models, diminishing-returns signal), with computation costing ~1 ms at d=50.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 31 of its 68 references that are in the map. First posted on arXiv 12 Jun 2026.

Builds on

31 of 68 references.

References · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  6. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  7. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  8. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  9. 2020Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?cited by 435
  10. 2018Deep Learning for Classical Japanese Literaturecited by 415
  11. 2018Meta-learning with differentiable closed-form solverscited by 395
  12. 2015An Introduction to Matrix Concentration Inequalitiescited by 391
19 more
  1. 2018Meta-Transfer Learning for Few-Shot Learningcited by 306
  2. 2019A Baseline for Few-Shot Image Classificationcited by 298
  3. 2018Few-Shot Learning via Embedding Adaptation with Set-to-Set Functionscited by 293
  4. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  5. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  6. 2017Emergence of Invariance and Disentanglement in Deep Representationscited by 213
  7. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 211
  8. 2020Provable Meta-Learning of Linear Representationscited by 161
  9. 2017Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practicecited by 132
  10. 2019Bayesian Batch Active Learning as Sparse Subset Approximationcited by 68
  11. 2019Learning-to-Learn Stochastic Gradient Descent with Biased Regularizationcited by 49
  12. 2023LaFTer: Label-Free Tuning of Zero-shot Classifier using Language and Unlabeled Image Collectionscited by 43
  13. 2023Double Descent Demystified: Identifying, Interpreting & Ablating the Sources of a Deep Learning Puzzlecited by 36
  14. 2024Towards Few-Shot Adaptation of Foundation Models via Multitask Finetuningcited by 31
  15. 2017Early stopping for kernel boosting algorithms: A general analysis with localized complexitiescited by 27
  16. 2022Bayesian Active Learning with Fully Bayesian Gaussian Processescited by 26
  17. 2024Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correctingcited by 26
  18. 2021Model, sample, and epoch-wise descents: exact solution of gradient flow in the random feature modelcited by 17
  19. 2024Covariance-based Space Regularization for Few-shot Class Incremental Learningcited by 2

Built on by

Not cited yet.