mlsanityData to 7 Oct 2026

Revealing the Utilized Rank of Subspaces of Learning in Neural Networks

arXiv 2407.04797 ↗First posted 5 Jul 2024cs.CV cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

In this work, we study how well the learned weights of a neural network utilize the space available to them. This notion is related to capacity, but additionally incorporates the interaction of the network architecture with the dataset. Most learned weights appear to be full rank, and are therefore not amenable to low rank decomposition. This deceptively implies that the weights are utilizing the entire space available to them. We propose a simple data-driven transformation that projects the weights onto the subspace where the data and the weight interact. This preserves the functional mapping of the layer and reveals its low rank structure. In our findings, we conclude that most models utilize a fraction of the available space. For instance, for ViTB-16 and ViTL-16 trained on ImageNet, the mean layer utilization is 35% and 20% respectively. Our transformation results in reducing the parameters to 50% and 25% respectively, while resulting in less than 0.2% accuracy drop after fine-tuning. We also show that self-supervised pre-training drives this utilization up to 70%, justifying its suitability for downstream tasks.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 31 of its 55 references that are in the map. First posted on arXiv 5 Jul 2024.

Builds on

31 of 55 references.

References · 31

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  8. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  10. 2016Wide Residual Networkscited by 3,057
  11. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  12. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
19 more
  1. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  2. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  3. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  4. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  5. 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 283
  6. 2023The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reductioncited by 120
  7. 2018Three Mechanisms of Weight Decay Regularizationcited by 112
  8. 2019Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobiancited by 67
  9. 2018Norm matters: efficient and accurate normalization schemes in deep networkscited by 62
  10. 2021Initialization and Regularization of Factorized Neural Layerscited by 58
  11. 2022Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equationscited by 42
  12. 2021Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decompositioncited by 33
  13. 2017Coordinating Filters for Faster Deep Neural Networkscited by 24
  14. 2016Recovery guarantee of weighted low-rank approximation via alternating minimizationcited by 20
  15. 2018Filter Distillation for Network Compressioncited by 17
  16. 2018A Low Effort Approach to Structured CNN Design Using PCAcited by 14
  17. 2024Provable Emergence of Deep Neural Collapse and Low-Rank Bias in L2-Regularized Nonlinear Networkscited by 10
  18. 2020Compressed Deep Networks: Goodbye SVD, Hello Robust Low-Rank Approximationcited by 8
  19. 2019Decomposable-Net: Scalable Low-Rank Compression for Neural Networkscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026Investigating Single-Block Recurrence in Vision Transformers for Image Recognitioncited by 3
  2. 2025On Information Geometry and Iterative Optimization in Model Compression: Operator Factorizationcited by 1
  3. 2026Training Crossroads for Recurrent Vision Transformers: Recurrence, Neural ODEs, and Deep Supervisioncited by 1
  4. 2026Learning Functional Subspaces for Neural Network Compressioncited by 0