mlsanityData to 7 Oct 2026

VNE: An Effective Method for Improving Deep Representation by Manipulating Eigenvalue Distribution

arXiv 2304.01434 ↗First posted 4 Apr 2023cs.CV cs.AI cs.LGcited by 17

Not on the map. Why?

Abstract (arXiv)

Since the introduction of deep learning, a wide scope of representation properties, such as decorrelation, whitening, disentanglement, rank, isotropy, and mutual information, have been studied to improve the quality of representation. However, manipulating such properties can be challenging in terms of implementational effectiveness and general applicability. To address these limitations, we propose to regularize von Neumann entropy~(VNE) of representation. First, we demonstrate that the mathematical formulation of VNE is superior in effectively manipulating the eigenvalues of the representation autocorrelation matrix. Then, we demonstrate that it is widely applicable in improving state-of-the-art algorithms or popular benchmark algorithms by investigating domain-generalization, meta-learning, self-supervised learning, and generative models. In addition, we formally establish theoretical connections with rank, disentanglement, and isotropy of representation. Finally, we provide discussions on the dimension control of VNE and the relationship with Shannon entropy. Code is available at: https://github.com/jaeill/CVPR23-VNE.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 69 of its 98 references that are in the map. First posted on arXiv 4 Apr 2023.

Builds on

69 of 98 references.

References · 69

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2017Mask R-CNNcited by 7,259
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  8. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  10. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  11. 2016Improved Techniques for Training GANscited by 2,938
  12. 2020Exploring Simple Siamese Representation Learningcited by 2,853
38 more
  1. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  2. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  3. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  4. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  5. 2016Matching Networks for One Shot Learningcited by 2,246
  6. 2019Invariant Risk Minimizationcited by 1,832
  7. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  8. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  9. 2019Contrastive Multiview Codingcited by 1,297
  10. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  11. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  12. 2017Deep Hashing Network for Unsupervised Domain Adaptationcited by 1,181
  13. 2020In Search of Lost Domain Generalizationcited by 1,042
  14. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  15. 2017Deeper, Broader and Artier Domain Generalizationcited by 983
  16. 2020What Makes for Good Views for Contrastive Learning?cited by 843
  17. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  18. 2020Out-of-Distribution Generalization via Risk Extrapolation (REx)cited by 737
  19. 2017SphereFace: Deep Hypersphere Embedding for Face Recognitioncited by 733
  20. 2019A Closer Look at Few-shot Classificationcited by 679
  21. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  22. 2018Recognition in Terra Incognitacited by 655
  23. 2017Deep Neural Networks as Gaussian Processescited by 617
  24. 2018Disentangling by Factorisingcited by 614
  25. 2019On Variational Bounds of Mutual Informationcited by 550
  26. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
  27. 2020Hard Negative Mixing for Contrastive Learningcited by 399
  28. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  29. 2019Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAMLcited by 359
  30. 2019On Mutual Information Maximization for Representation Learningcited by 315
  31. 2016NIPS 2016 Tutorial: Generative Adversarial Networkscited by 284
  32. 2017NormFace: L2 Hypersphere Embedding for Face Verificationcited by 237
  33. 2020What Should Not Be Contrastive in Contrastive Learningcited by 224
  34. 2021Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalizationcited by 220
  35. 2017Emergence of Invariance and Disentanglement in Deep Representationscited by 213
  36. 2017All-but-the-Top: Simple and Effective Postprocessing for Word Representationscited by 205
  37. 2019Scaling and Benchmarking Self-Supervised Visual Representation Learningcited by 201
  38. 2021Whitening Sentence Representations for Better Semantics and Faster Retrievalcited by 198

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 9; 2025: 4.

17 papers.

Sort:

By area:

  1. 2024Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similaritiescited by 156
  2. 2023Matrix Information Theory for Self-Supervised Learningcited by 21
  3. 2024HYPO: Hyperspherical Out-of-Distribution Generalizationcited by 13
  4. 2023CIFAR-10-Warehouse: Broad and More Realistic Testbeds in Model Generalization Analysiscited by 12
  5. 2024Rethinking the Representation in Federated Unsupervised Learning with Non-IID Datacited by 12
  6. 2024AHA: Human-Assisted Out-of-Distribution Generalization and Detectioncited by 7
  7. 2024Out-of-Distribution Learning with Human Feedbackcited by 6
  8. 2024Learning to Embed Distributions via Maximum Kernel Entropycited by 3
  9. 2025AdaDim: Dimensionality Adaptation for SSL Representational Dynamicscited by 3
  10. 2023FroSSL: Frobenius Norm Minimization for Efficient Multiview Self-Supervised Learningcited by 2
  11. 2023WeedCLR: Weed Contrastive Learning through Visual Representations with Class-Optimized Loss in Long-Tailed Datasetscited by 1
  12. 2024Improving Forward Compatibility in Class Incremental Learning by Increasing Representation Rank and Feature Richnesscited by 1
5 more
  1. 2024Exploring Information-Theoretic Metrics Associated with Neural Collapse in Supervised Trainingcited by 1
  2. 2025Countering Multi-modal Representation Collapse through Rank-targeted Fusioncited by 1
  3. 2024Unveiling Key Aspects of Fine-Tuning in Sentence Embeddings: A Representation Rank Analysiscited by 0
  4. 2025Vision and Language Integration for Domain Generalizationcited by 0
  5. 2025NexusFlow: Unifying Disparate Tasks under Partial Supervision via Invertible Flow Networkscited by 0