# Fix your classifier: the marginal value of training the last weight layer

arXiv: https://arxiv.org/abs/1801.04540 · HTML: https://mlsanity.com/paper/1801.04540/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jan 2018.

## Built on by (most cited first, 30 of 43)

- [A Geometric Analysis of Neural Collapse with Unconstrained Features](https://mlsanity.com/paper/2105.02375/) (2021)
- [Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?](https://mlsanity.com/paper/2203.09081/) (2022)
- [Constrained Few-shot Class-incremental Learning](https://mlsanity.com/paper/2203.16588/) (2022)
- [On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)](https://mlsanity.com/paper/2102.12470/) (2021)
- [nGPT: Normalized Transformer with Representation Learning on the Hypersphere](https://mlsanity.com/paper/2410.01131/) (2024)
- [Dissecting Supervised Contrastive Learning](https://mlsanity.com/paper/2102.08817/) (2021)
- [No One Representation to Rule Them All: Overlapping Features of Training Methods](https://mlsanity.com/paper/2110.12899/) (2021)
- [Self-Supervised Classification Network](https://mlsanity.com/paper/2103.10994/) (2021)
- [Regular Polytope Networks](https://mlsanity.com/paper/2103.15632/) (2021)
- [No Reason for No Supervision: Improved Generalization in Supervised Models](https://mlsanity.com/paper/2206.15369/) (2022)
- [On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decay](https://mlsanity.com/paper/2106.15739/) (2021)
- [SphereFed: Hyperspherical Federated Learning](https://mlsanity.com/paper/2207.09413/) (2022)
- [CoReS: Compatible Representations via Stationarity](https://mlsanity.com/paper/2111.07632/) (2021)
- [Neural Collapse Terminus: A Unified Solution for Class Incremental Learning and Its Variants](https://mlsanity.com/paper/2308.01746/) (2023)
- [Guiding Neural Collapse: Optimising Towards the Nearest Simplex Equiangular Tight Frame](https://mlsanity.com/paper/2411.01248/) (2024)
- [Inducing Neural Collapse to a Fixed Hierarchy-Aware Frame for Reducing Mistake Severity](https://mlsanity.com/paper/2303.05689/) (2023)
- [Generalized Learning Vector Quantization for Classification in Randomized Neural Networks and Hyperdimensional Computing](https://mlsanity.com/paper/2106.09821/) (2021)
- [Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Survey](https://mlsanity.com/paper/2205.08099/) (2022)
- [Towards Continual Learning Desiderata via HSIC-Bottleneck Orthogonalization and Equiangular Embedding](https://mlsanity.com/paper/2401.09067/) (2024)
- [Engineering the Neural Collapse Geometry of Supervised-Contrastive Loss](https://mlsanity.com/paper/2310.00893/) (2023)
- [Reset It and Forget It: Relearning Last-Layer Weights Improves Continual and Transfer Learning](https://mlsanity.com/paper/2310.07996/) (2023)
- [Slot Machines: Discovering Winning Combinations of Random Weights in Neural Networks](https://mlsanity.com/paper/2101.06475/) (2021)
- [Ex uno plures: Splitting One Model into an Ensemble of Subnetworks](https://mlsanity.com/paper/2106.04767/) (2021)
- [Maximally Compact and Separated Features with Regular Polytope Networks](https://mlsanity.com/paper/2301.06116/) (2023)
- [Factorizers for Distributed Sparse Block Codes](https://mlsanity.com/paper/2303.13957/) (2023)
- [Pairwise Similarity Learning is SimPLE](https://mlsanity.com/paper/2310.09449/) (2023)
- [Out-of-Distribution Detection with Prototypical Outlier Proxy](https://mlsanity.com/paper/2412.16884/) (2024)
- [Hyperspherical Classification with Dynamic Label-to-Prototype Assignment](https://mlsanity.com/paper/2403.16937/) (2024)
- [Rethinking Decoders for Transformer-based Semantic Segmentation: A Compression Perspective](https://mlsanity.com/paper/2411.03033/) (2024)
- [LogAvgExp Provides a Principled and Performant Global Pooling Operator](https://mlsanity.com/paper/2111.01742/) (2021)

## Abstract (arXiv)

Neural networks are commonly used as models for classification for a wide variety of tasks. Typically, a learned affine transformation is placed at the end of such models, yielding a per-class value used for classification. This classifier can have a vast number of parameters, which grows linearly with the number of possible classes, thus requiring increasingly more resources. In this work we argue that this classifier can be fixed, up to a global scale constant, with little or no loss of accuracy for most tasks, allowing memory and computational benefits. Moreover, we show that by initializing the classifier with a Hadamard matrix we can speed up inference as well. We discuss the implications for current understanding of neural network models.