mlsanityData to 7 Oct 2026

Fix your classifier: the marginal value of training the last weight layer

arXiv 1801.04540 ↗First posted 14 Jan 2018cs.LG cs.CV stat.MLcited by 43

Not on the map. Why?

Abstract (arXiv)

Neural networks are commonly used as models for classification for a wide variety of tasks. Typically, a learned affine transformation is placed at the end of such models, yielding a per-class value used for classification. This classifier can have a vast number of parameters, which grows linearly with the number of possible classes, thus requiring increasingly more resources. In this work we argue that this classifier can be fixed, up to a global scale constant, with little or no loss of accuracy for most tasks, allowing memory and computational benefits. Moreover, we show that by initializing the classifier with a Hadamard matrix we can speed up inference as well. We discuss the implications for current understanding of neural network models.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Jan 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 7; 2023: 8; 2024: 6; 2025: 1; 2026: 8.

43 papers.

Sort:

  1. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  2. 2022Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?cited by 100
  3. 2022Constrained Few-shot Class-incremental Learningcited by 93
  4. 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
  5. 2024nGPT: Normalized Transformer with Representation Learning on the Hyperspherecited by 68
  6. 2021Dissecting Supervised Contrastive Learningcited by 64
  7. 2021No One Representation to Rule Them All: Overlapping Features of Training Methodscited by 57
  8. 2021Self-Supervised Classification Networkcited by 27
  9. 2021Regular Polytope Networkscited by 24
  10. 2022No Reason for No Supervision: Improved Generalization in Supervised Modelscited by 23
  11. 2021On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decaycited by 20
  12. 2022SphereFed: Hyperspherical Federated Learningcited by 19
31 more
  1. 2021CoReS: Compatible Representations via Stationaritycited by 14
  2. 2023Neural Collapse Terminus: A Unified Solution for Class Incremental Learning and Its Variantscited by 14
  3. 2024Guiding Neural Collapse: Optimising Towards the Nearest Simplex Equiangular Tight Framecited by 14
  4. 2023Inducing Neural Collapse to a Fixed Hierarchy-Aware Frame for Reducing Mistake Severitycited by 13
  5. 2021Generalized Learning Vector Quantization for Classification in Randomized Neural Networks and Hyperdimensional Computingcited by 12
  6. 2022Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Surveycited by 12
  7. 2024Towards Continual Learning Desiderata via HSIC-Bottleneck Orthogonalization and Equiangular Embeddingcited by 9
  8. 2023Engineering the Neural Collapse Geometry of Supervised-Contrastive Losscited by 8
  9. 2023Reset It and Forget It: Relearning Last-Layer Weights Improves Continual and Transfer Learningcited by 6
  10. 2021Slot Machines: Discovering Winning Combinations of Random Weights in Neural Networkscited by 5
  11. 2021Ex uno plures: Splitting One Model into an Ensemble of Subnetworkscited by 5
  12. 2023Maximally Compact and Separated Features with Regular Polytope Networkscited by 5
  13. 2023Factorizers for Distributed Sparse Block Codescited by 4
  14. 2023Pairwise Similarity Learning is SimPLEcited by 4
  15. 2024Out-of-Distribution Detection with Prototypical Outlier Proxycited by 4
  16. 2024Hyperspherical Classification with Dynamic Label-to-Prototype Assignmentcited by 3
  17. 2024Rethinking Decoders for Transformer-based Semantic Segmentation: A Compression Perspectivecited by 3
  18. 2021LogAvgExp Provides a Principled and Performant Global Pooling Operatorcited by 2
  19. 2022Reinforcement Learning with Large Action Spaces for Neural Machine Translationcited by 2
  20. 2023The Role of Codeword-to-Class Assignments in Error-Correcting Codes: An Empirical Studycited by 2
  21. 2025GIF: Generative Inspiration for Face Recognition at Scalecited by 2
  22. 2022The Fixed Sub-Center: A Better Way to Capture Data Complexitycited by 1
  23. 2026What Do Scan-Derived Class Prototypes Add? Disentangling Supervision, Prototype Content and Query Protocol in Recognition over Frozen Foundation Featurescited by 1
  24. 2021Classifier Crafting: Turn Your ConvNet into a Zero-Shot Learner!cited by 0
  25. 2026Simplex Deep Linear Discriminant Analysiscited by 0
  26. 2026RST: certifying and constructing prescribed information in variational autoencoderscited by 0
  27. 2026Adversarial Attack and Disturbance Detection by Hadamard-Coded Output Representations for Object Detection and Semantic Segmentationcited by 0
  28. 2026Contrastive-Collapsed Loss for Flexible and Geometrically Optimal Embeddings and Faster Convergencecited by 0
  29. 2026Federated Lightweight Fine-Tuningcited by 0
  30. 2026OPAL: Orthonormal Prototype Alignment Learning for Interpretable Image Classificationcited by 0
  31. 2026SOLO: Pretraining Billion-Parameter Language Models with Shared-Output Local Learningcited by 0