mlsanityData to 14 Oct 2026

Fisher Rank Inflation: A Spectral Signature of Memorization under Label Noise

arXiv 2607.12438 ↗First posted 14 Jul 2026cs.LG stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Deep networks trained with label noise often learn clean structure before memorizing corrupted labels. We show that this transition leaves a spectral signature in the centered scatter of per-example last-layer gradients. Its effective rank transiently expands during memorization and contracts after corrupted labels are fit. We call this phenomenon Fisher Rank Inflation. Corrupted labels increase effective rank by injecting spectral mass into low-energy or previously unused eigendirections, increasing the entropy of the gradient spectrum. We derive a first-order leave-one-out attribution formula, identify conditions under which corrupted examples contribute more strongly than clean examples, and explain why attribution signals weaken once the normalized Fisher-gradient spectrum stabilizes. We test these predictions on CIFAR-10, CIFAR-100, and CIFAR-10N using SmallCNN, ResNet18, and Vision Transformers. Across settings, Fisher effective rank exhibits a consistent inflation--collapse trajectory aligned with memorization. At peak-rank checkpoints, corrupted examples are enriched among the highest rank-contributing samples, with top-100 noisy fractions from (69.2%) to (96.2%) across five-seed synthetic-corruption experiments and (94.4%±1.9%) on CIFAR-10N. First-order spectral attribution closely matches exact leave-one-out contributions in convolutional models and remains enriched in the Vision Transformer. Peak effective rank increases monotonically with corruption severity, from (28.88±1.95) under clean training to (97.09±1.78) at (60%) corruption. In several settings, the retrospectively identified onset of rank inflation precedes observable test degradation. These results establish Fisher Rank Inflation as a spectral signature connecting corrupted-example enrichment, corruption severity, and the transition from structure learning to memorization.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 37 of its 40 references that are in the map. First posted on arXiv 14 Jul 2026.

Builds on

37 of 40 references.

References · 37

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  3. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  4. 2018On the Spectral Bias of Neural Networkscited by 1,341
  5. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,059
  6. 2017A Closer Look at Memorization in Deep Networkscited by 942
  7. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  8. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  9. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  10. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
  11. 2020DivideMix: Learning with Noisy Labels as Semi-supervised Learningcited by 692
  12. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 678
25 more
  1. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
  2. 2016Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approachcited by 665
  3. 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 629
  4. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
  5. 2019Symmetric Cross Entropy for Robust Learning with Noisy Labelscited by 500
  6. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 497
  7. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 460
  8. 2020Early-Learning Regularization Prevents Memorization of Noisy Labelscited by 416
  9. 2019Confident Learning: Estimating Uncertainty in Dataset Labelscited by 413
  10. 2019How does Disagreement Help Generalization against Label Corruption?cited by 390
  11. 2019Unsupervised Label Noise Modeling and Loss Correctioncited by 328
  12. 2014Training Deep Neural Networks on Noisy Labels with Bootstrappingcited by 322
  13. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 290
  14. 2020Combating noisy labels by agreement: A joint training method with co-regularizationcited by 289
  15. 2020Identifying Mislabeled Data using the Area Under the Margin Rankingcited by 261
  16. 2020Influence Functions in Deep Learning Are Fragilecited by 252
  17. 2018Representer Point Selection for Explaining Deep Neural Networkscited by 187
  18. 2017CleanNet: Transfer Learning for Scalable Image Classifier Training with Label Noisecited by 181
  19. 2017Deep Learning is Robust to Massive Label Noisecited by 165
  20. 2018Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasiscited by 146
  21. 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 141
  22. 2020FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debuggingcited by 120
  23. 2019Traditional and Heavy-Tailed Self Regularization in Neural Network Modelscited by 105
  24. 2018Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approachcited by 91
  25. 2019Emergent properties of the local geometry of neural loss landscapescited by 46

Built on by

Not cited yet.