Get the Best of Both Worlds: Improving Accuracy and Transferability by Grassmann Class Representation
Not on the map. Why?
Abstract (arXiv)
We generalize the class vectors found in neural networks to linear subspaces (i.e.~points in the Grassmann manifold) and show that the Grassmann Class Representation (GCR) enables the simultaneous improvement in accuracy and feature transferability. In GCR, each class is a subspace and the logit is defined as the norm of the projection of a feature onto the class subspace. We integrate Riemannian SGD into deep learning frameworks such that class subspaces in a Grassmannian are jointly optimized with the rest model parameters. Compared to the vector form, the representative capability of subspaces is more powerful. We show that on ImageNet-1K, the top-1 error of ResNet50-D, ResNeXt50, Swin-T and Deit3-S are reduced by 5.6%, 4.5%, 3.0% and 3.5%, respectively. Subspaces also provide freedom for features to vary and we observed that the intra-class feature variability grows when the subspace dimension increases. Consequently, we found the quality of GCR features is better for downstream tasks. For ResNet50-D, the average linear transfer accuracy across 6 datasets improves from 77.98% to 79.70% compared to the strong baseline of vanilla softmax. For Swin-T, it improves from 81.5% to 83.4% and for Deit3, it improves from 73.8% to 81.4%. With these encouraging results, we believe that more applications could benefit from the Grassmann class representation. Code is released at https://github.com/innerlee/GCR.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 30 of its 58 references that are in the map. First posted on arXiv 3 Aug 2023.
Builds on
30 of 58 references.
References · 30
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
- 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
- 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
- 2019When Does Label Smoothing Help?cited by 988
- 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
- 2018Bag of Tricks for Image Classification with Convolutional Neural Networkscited by 431
- 2022DeiT III: Revenge of the ViTcited by 419
- 2022ViM: Out-Of-Distribution with Virtual-logit Matchingcited by 405
- 2021ResNet strikes back: An improved training procedure in timmcited by 368
18 more
- 2020Are we done with ImageNet?cited by 333
- 2015Unitary Evolution Recurrent Neural Networkscited by 294
- 2018Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometrycited by 265
- 2019Fixing the train-test resolution discrepancycited by 209
- 2018Riemannian Adaptive Optimization Methodscited by 194
- 2019SoftTriple Loss: Deep Metric Learning Without Triplet Samplingcited by 163
- 2020Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reductioncited by 142
- 2019Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Groupcited by 134
- 2019Orthogonal Convolutional Neural Networkscited by 123
- 2019Trivializations for Gradient-Based Optimization on Manifoldscited by 79
- 2020Why Do Better Loss Functions Lead to Less Transferable Features?cited by 73
- 2017All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulationcited by 58
- 2020Deep Isometric Learning for Visual Recognitioncited by 39
- 2017Learning Spread-out Local Feature Descriptorscited by 34
- 2016Generalized BackPropagation, 'Etude De Cas: Orthogonalitycited by 17
- 2018Scalable Deep k-Subspace Clusteringcited by 14
- 2013R3MC: A Riemannian three-factor algorithm for low-rank matrix completioncited by 11
- 2017A dual framework for low-rank tensor completioncited by 9
Built on by
Citing papers per year: 2025: 3.
3 papers.