# Maximum Class Separation as Inductive Bias in One Matrix

arXiv: https://arxiv.org/abs/2206.08704 · HTML: https://mlsanity.com/paper/2206.08704/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 70 references that are in the map.
- First posted on arXiv 17 Jun 2022.

## Builds on (most cited first, 30 of 55)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds](https://mlsanity.com/paper/1802.08219/) (2018)
- [SphereFace: Deep Hypersphere Embedding for Face Recognition](https://mlsanity.com/paper/1704.08063/) (2017)
- [Prevalence of Neural Collapse during the terminal phase of deep learning training](https://mlsanity.com/paper/2008.08186/) (2020)
- [Large-Scale Long-Tailed Recognition in an Open World](https://mlsanity.com/paper/1904.05160/) (2019)
- [Directional Message Passing for Molecular Graphs](https://mlsanity.com/paper/2003.03123/) (2020)
- [Open-Set Recognition: a Good Closed-Set Classifier is All You Need?](https://mlsanity.com/paper/2110.06207/) (2021)
- [Large-Margin Softmax Loss for Convolutional Neural Networks](https://mlsanity.com/paper/1612.02295/) (2016)
- [Harmonic Networks: Deep Translation and Rotation Equivariance](https://mlsanity.com/paper/1612.04642/) (2016)
- [Gauge Equivariant Convolutional Networks and the Icosahedral CNN](https://mlsanity.com/paper/1902.04615/) (2019)
- [Cormorant: Covariant Molecular Neural Networks](https://mlsanity.com/paper/1906.04015/) (2019)
- [Geometric and Physical Quantities Improve E(3) Equivariant Message Passing](https://mlsanity.com/paper/2110.02905/) (2021)
- [A Geometric Analysis of Neural Collapse with Unconstrained Features](https://mlsanity.com/paper/2105.02375/) (2021)
- [ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias](https://mlsanity.com/paper/2106.03348/) (2021)
- [Improving Calibration for Long-Tailed Recognition](https://mlsanity.com/paper/2104.00466/) (2021)
- [Ab-Initio Solution of the Many-Electron Schr"odinger Equation with Deep Neural Networks](https://mlsanity.com/paper/1909.02487/) (2019)
- [Graph Convolutional Reinforcement Learning](https://mlsanity.com/paper/1810.09202/) (2018)
- [MDP Homomorphic Networks: Group Symmetries in Reinforcement Learning](https://mlsanity.com/paper/2006.16908/) (2020)

## Built on by (most cited first, 21 of 21)

- [Hyperbolic Deep Learning in Computer Vision: A Survey](https://mlsanity.com/paper/2305.06611/) (2023)
- [Use Your Head: Improving Long-Tail Video Recognition](https://mlsanity.com/paper/2304.01143/) (2023)
- [Position: Beyond Euclidean -- Foundation Models Should Embrace Non-Euclidean Geometries](https://mlsanity.com/paper/2504.08896/) (2025)
- [Balancing the Scales: A Theoretical and Algorithmic Framework for Learning from Imbalanced Data](https://mlsanity.com/paper/2502.10381/) (2025)
- [Forgetting, Ignorance or Myopia: Revisiting Key Challenges in Online Continual Learning](https://mlsanity.com/paper/2409.19245/) (2024)
- [Combating Representation Learning Disparity with Geometric Harmonization](https://mlsanity.com/paper/2310.17622/) (2023)
- [Hyperbolic Random Forests](https://mlsanity.com/paper/2308.13279/) (2023)
- [Improved Balanced Classification with Theoretically Grounded Loss Functions](https://mlsanity.com/paper/2512.23947/) (2025)
- [A Tale of Two Classes: Adapting Supervised Contrastive Learning to Binary Imbalanced Datasets](https://mlsanity.com/paper/2503.17024/) (2025)
- [Pairwise Similarity Learning is SimPLE](https://mlsanity.com/paper/2310.09449/) (2023)
- [On the Dynamics Under the Unhinged Loss and Beyond](https://mlsanity.com/paper/2312.07841/) (2023)
- [Hyperspherical Classification with Dynamic Label-to-Prototype Assignment](https://mlsanity.com/paper/2403.16937/) (2024)
- [All-around Neural Collapse for Imbalanced Classification](https://mlsanity.com/paper/2408.07253/) (2024)
- [Adversarial Attacks on Hyperbolic Networks](https://mlsanity.com/paper/2412.01495/) (2024)
- [Understanding Open-Set Recognition by Jacobian Norm and Inter-Class Separation](https://mlsanity.com/paper/2209.11436/) (2022)
- [GIF: Generative Inspiration for Face Recognition at Scale](https://mlsanity.com/paper/2505.03012/) (2025)
- [Optimized Deferral for Imbalanced Settings](https://mlsanity.com/paper/2604.27723/) (2026)
- [A Coding-Theoretic Analysis of Hyperspherical Prototypical Learning Geometry](https://mlsanity.com/paper/2407.07664/) (2024)
- [Face-Voice Association with Inductive Bias for Maximum Class Separation](https://mlsanity.com/paper/2601.13651/) (2026)
- [Local learning for stable backpropagation-free neural network training towards physical learning](https://mlsanity.com/paper/2603.24790/) (2026)
- [Lightweight Uncertainty Quantification with Simplex Semantic Segmentation for Terrain Traversability](https://mlsanity.com/paper/2407.13392/) (2024)

## Abstract (arXiv)

Maximizing the separation between classes constitutes a well-known inductive bias in machine learning and a pillar of many traditional algorithms. By default, deep networks are not equipped with this inductive bias and therefore many alternative solutions have been proposed through differential optimization. Current approaches tend to optimize classification and separation jointly: aligning inputs with class vectors and separating class vectors angularly. This paper proposes a simple alternative: encoding maximum separation as an inductive bias in the network by adding one fixed matrix multiplication before computing the softmax activations. The main observation behind our approach is that separation does not require optimization but can be solved in closed-form prior to training and plugged into a network. We outline a recursive approach to obtain the matrix consisting of maximally separable vectors for any number of classes, which can be added with negligible engineering effort and computational overhead. Despite its simple nature, this one matrix multiplication provides real impact. We show that our proposal directly boosts classification, long-tailed recognition, out-of-distribution detection, and open-set recognition, from CIFAR to ImageNet. We find empirically that maximum separation works best as a fixed bias; making the matrix learnable adds nothing to the performance. The closed-form implementation and code to reproduce the experiments are available on github.