mlsanityData to 7 Oct 2026

Learning Representation from Neural Fisher Kernel with Low-rank Approximation

arXiv 2202.01944 ↗First posted 4 Feb 2022cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

In this paper, we study the representation of neural networks from the view of kernels. We first define the Neural Fisher Kernel (NFK), which is the Fisher Kernel applied to neural networks. We show that NFK can be computed for both supervised and unsupervised learning models, which can serve as a unified tool for representation extraction. Furthermore, we show that practical NFKs exhibit low-rank structures. We then propose an efficient algorithm that computes a low rank approximation of NFK, which scales to large datasets and networks. We show that the low-rank approximation of NFKs derived from unsupervised generative models and supervised learning models gives rise to high-quality compact representations of data, achieving competitive results on a variety of machine learning tasks.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 49 of its 78 references that are in the map. First posted on arXiv 4 Feb 2022.

Builds on

49 of 78 references.

References · 49

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  7. 2013Auto-Encoding Variational Bayescited by 3,980
  8. 2016Wide Residual Networkscited by 3,057
  9. 2016Improved Techniques for Training GANscited by 2,938
  10. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  11. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  12. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
37 more
  1. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,390
  2. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  3. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  4. 2016Temporal Ensembling for Semi-Supervised Learningcited by 1,054
  5. 2014NICE: Non-linear Independent Components Estimationcited by 1,052
  6. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 918
  7. 2016Pixel Recurrent Neural Networkscited by 814
  8. 2019Contrastive Representation Distillationcited by 734
  9. 2019Self-supervised Visual Feature Learning with Deep Neural Networks: A Surveycited by 656
  10. 2017Deep Neural Networks as Gaussian Processescited by 617
  11. 2019On Variational Bounds of Mutual Informationcited by 550
  12. 2019Similarity-Preserving Knowledge Distillationcited by 507
  13. 2017Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning resultscited by 470
  14. 2013Do Deep Nets Really Need to be Deep?cited by 466
  15. 2015Importance Weighted Autoencoderscited by 454
  16. 2016Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learningcited by 441
  17. 2019Your Classifier is Secretly an Energy Based Model and You Should Treat it Like Onecited by 402
  18. 2019Variational Information Distillation for Knowledge Transfercited by 321
  19. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  20. 2020Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Imagescited by 261
  21. 2019Task2Vec: Task Embedding for Meta-Learningcited by 197
  22. 2018To understand deep learning we need to understand kernel learningcited by 186
  23. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  24. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  25. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  26. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
  27. 2020Traces of Class/Cross-Class Structure Pervade Deep Learning Spectracited by 94
  28. 2019AET vs. AED: Unsupervised Representation Learning by Auto-Encoding Transformations rather than Datacited by 77
  29. 2020On the Similarity between the Laplace and Neural Tangent Kernelscited by 76
  30. 2020Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHScited by 74
  31. 2020Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Samplingcited by 71
  32. 2020Gradients as Features for Deep Representation Learningcited by 30
  33. 2015Steps Toward Deep Kernel Methods from Infinite Neural Networkscited by 28
  34. 2020Understanding Approximate Fisher Information for Fast Convergence of Natural Gradient Descent in Wide Neural Networkscited by 22
  35. 2017Calibrating Energy-based Generative Adversarial Networkscited by 13
  36. 2019Adversarial Fisher Vectors for Unsupervised Representation Learningcited by 5
  37. 2020Learning Structured Latent Factors from Dependent Data:A Generative Model Framework from Information-Theoretic Perspectivecited by 5

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 0; 2025: 2.

4 papers.

  1. 2023Supervision Complexity and its Role in Knowledge Distillationcited by 19
  2. 2025Flexible Language Modeling in Continuous Space with Transformer-based Autoregressive Flowscited by 8
  3. 2023Low-Dimensional Gradient Helps Out-of-Distribution Detectioncited by 6
  4. 2025KnowledgeSmith: Uncovering Knowledge Updating in LLMs with Model Editing and Unlearningcited by 3