# The Geometry of Saturation: Effective Rank Predicts When Labels Stop Helping in Few-Shot Classification

arXiv: https://arxiv.org/abs/2606.24903 · HTML: https://mlsanity.com/paper/2606.24903/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Sparse autoencoder training and evaluation, one of 13 directions in Mechanistic interpretability.
- Builds on 31 of its 68 references that are in the map.
- First posted on arXiv 12 Jun 2026.

## Placed in

- Mechanistic interpretability / [Sparse autoencoder training and evaluation](https://mlsanity.com/mechinterp/sparse-autoencoder-training-and-evaluation/)

## Builds on (most cited first, 30 of 31)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Prevalence of Neural Collapse during the terminal phase of deep learning training](https://mlsanity.com/paper/2008.08186/) (2020)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?](https://mlsanity.com/paper/2003.11539/) (2020)
- [Deep Learning for Classical Japanese Literature](https://mlsanity.com/paper/1812.01718/) (2018)
- [An Introduction to Matrix Concentration Inequalities](https://mlsanity.com/paper/1501.01571/) (2015)
- [Meta-learning with differentiable closed-form solvers](https://mlsanity.com/paper/1805.08136/) (2018)
- [Meta-Transfer Learning for Few-Shot Learning](https://mlsanity.com/paper/1812.02391/) (2018)
- [A Baseline for Few-Shot Image Classification](https://mlsanity.com/paper/1909.02729/) (2019)
- [Few-Shot Learning via Embedding Adaptation with Set-to-Set Functions](https://mlsanity.com/paper/1812.03664/) (2018)
- [An Investigation into Neural Net Optimization via Hessian Eigenvalue Density](https://mlsanity.com/paper/1901.10159/) (2019)
- [Empirical Analysis of the Hessian of Over-Parametrized Neural Networks](https://mlsanity.com/paper/1706.04454/) (2017)
- [Emergence of Invariance and Disentanglement in Deep Representations](https://mlsanity.com/paper/1706.01350/) (2017)
- [Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning](https://mlsanity.com/paper/1810.01075/) (2018)
- [Provable Meta-Learning of Linear Representations](https://mlsanity.com/paper/2002.11684/) (2020)
- [Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice](https://mlsanity.com/paper/1711.04735/) (2017)
- [Bayesian Batch Active Learning as Sparse Subset Approximation](https://mlsanity.com/paper/1908.02144/) (2019)
- [Learning-to-Learn Stochastic Gradient Descent with Biased Regularization](https://mlsanity.com/paper/1903.10399/) (2019)
- [LaFTer: Label-Free Tuning of Zero-shot Classifier using Language and Unlabeled Image Collections](https://mlsanity.com/paper/2305.18287/) (2023)
- [Double Descent Demystified: Identifying, Interpreting & Ablating the Sources of a Deep Learning Puzzle](https://mlsanity.com/paper/2303.14151/) (2023)
- [Towards Few-Shot Adaptation of Foundation Models via Multitask Finetuning](https://mlsanity.com/paper/2402.15017/) (2024)
- [Early stopping for kernel boosting algorithms: A general analysis with localized complexities](https://mlsanity.com/paper/1707.01543/) (2017)
- [Bayesian Active Learning with Fully Bayesian Gaussian Processes](https://mlsanity.com/paper/2205.10186/) (2022)
- [Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correcting](https://mlsanity.com/paper/2410.19294/) (2024)
- [Model, sample, and epoch-wise descents: exact solution of gradient flow in the random feature model](https://mlsanity.com/paper/2110.11805/) (2021)

## Abstract (arXiv)

Few-shot label acquisition lacks a label-free signal for when additional labels cease to improve accuracy: existing stopping criteria either require a held-out validation set (violating the few-shot premise) or rely on theoretically ungrounded heuristics, so we introduce the spectral saturation index S(K)=erank(hatSigmaW(K))/K, the exponential spectral entropy of the pooled within-class covariance normalized by per-class support size K, which measures the exploration rate per label and falls below a fixed threshold τ=0.02 once the explored spectral subspace saturates and marginal accuracy gains vanish; across 49 real tasks (binary, 5-way, 10-way) and three frozen backbones (PCA-50, CLIP ViT-B/32, DINOv2 ViT-S/14), S(K) correlates strongly with the marginal gain on doubling the support set (ρpool=0.6366, p=2.9×10-57, cluster-bootstrap 95% CI [0.551,0.720]), a fixed τ=0.02 classifies stop/continue decisions with cluster-bootstrap AUC=0.787 (95% CI [0.713,0.860]) with high recall on meaningful gains (Delta A>1%), and a partial correlation controlling for log K yields ρpartial=0.324 (p=1.65×10-13), confirming S(K) carries spectral information beyond shared K-dependence; theory predicts this from first principles, since the population effective rank sets the saturation scale Ksat≈erank(SigmaW)/τ, τ=0.02 sits at the boundary between the first and second descent (Nakkiran et al., 2021), and O(1/K) bias in the sample effective rank explains the small-K hump in S(K); for unregularized linear probes (C=∞), practitioners should halt when S(K)<0.02 (PCA-50, hard stop) or monitor S(K) dropping from ~0.3→0.05 (foundation models, diminishing-returns signal), with computation costing ~1 ms at d=50.