# Feature Incay for Representation Regularization

arXiv: https://arxiv.org/abs/1705.10284 · HTML: https://mlsanity.com/paper/1705.10284/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 May 2017.

## Built on by (most cited first, 9 of 9)

- [MagFace: A Universal Representation for Face Recognition and Quality Assessment](https://mlsanity.com/paper/2103.06627/) (2021)
- [Understanding the Feature Norm for Out-of-Distribution Detection](https://mlsanity.com/paper/2310.05316/) (2023)
- [Regular Polytope Networks](https://mlsanity.com/paper/2103.15632/) (2021)
- [BoundaryFace: A mining framework with noise label self-correction for Face Recognition](https://mlsanity.com/paper/2210.04567/) (2022)
- [Optimization Induced Equilibrium Networks](https://mlsanity.com/paper/2105.13228/) (2021)
- [Maximally Compact and Separated Features with Regular Polytope Networks](https://mlsanity.com/paper/2301.06116/) (2023)
- [Feature-Balanced Loss for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2305.10772/) (2023)
- [A Softmax-free Loss Function Based on Predefined Optimal-distribution of Latent Features for Deep Learning Classifier](https://mlsanity.com/paper/2111.15449/) (2021)
- [Rapid detection and recognition of whole brain activity in a freely behaving Caenorhabditis elegans](https://mlsanity.com/paper/2109.10474/) (2021)

## Abstract (arXiv)

Softmax loss is widely used in deep neural networks for multi-class classification, where each class is represented by a weight vector, a sample is represented as a feature vector, and the feature vector has the largest projection on the weight vector of the correct category when the model correctly classifies a sample. To ensure generalization, weight decay that shrinks the weight norm is often used as regularizer. Different from traditional learning algorithms where features are fixed and only weights are tunable, features are also tunable as representation learning in deep learning. Thus, we propose feature incay to also regularize representation learning, which favors feature vectors with large norm when the samples can be correctly classified. With the feature incay, feature vectors are further pushed away from the origin along the direction of their corresponding weight vectors, which achieves better inter-class separability. In addition, the proposed feature incay encourages intra-class compactness along the directions of weight vectors by increasing the small feature norm faster than the large ones. Empirical results on MNIST, CIFAR10 and CIFAR100 demonstrate feature incay can improve the generalization ability.