# Noisy Softmax: Improving the Generalization Ability of DCNN via Postponing the Early Softmax Saturation

arXiv: https://arxiv.org/abs/1708.03769 · HTML: https://mlsanity.com/paper/1708.03769/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Aug 2017.

## Built on by (most cited first, 21 of 21)

- [Long-tailed Visual Recognition via Gaussian Clouded Logit Adjustment](https://mlsanity.com/paper/2305.11733/) (2023)
- [Reducing Information Bottleneck for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2110.06530/) (2021)
- [OPOM: Customized Invisible Cloak towards Face Privacy Protection](https://mlsanity.com/paper/2205.11981/) (2022)
- [Rethinking Exemplars for Continual Semantic Segmentation in Endoscopy Scenes: Entropy-based Mini-Batch Pseudo-Replay](https://mlsanity.com/paper/2308.14100/) (2023)
- [Curse of Dimensionality for TSK Fuzzy Neural Networks: Explanation and Solutions](https://mlsanity.com/paper/2102.04271/) (2021)
- [Balanced Destruction-Reconstruction Dynamics for Memory-replay Class Incremental Learning](https://mlsanity.com/paper/2308.01698/) (2023)
- [MANO: Exploiting Matrix Norm for Unsupervised Accuracy Estimation Under Distribution Shifts](https://mlsanity.com/paper/2405.18979/) (2024)
- [Adjusting Logit in Gaussian Form for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2305.10648/) (2023)
- [Recent Advances in Deep Learning Techniques for Face Recognition](https://mlsanity.com/paper/2103.10492/) (2021)
- [DAIL: Dataset-Aware and Invariant Learning for Face Recognition](https://mlsanity.com/paper/2101.05419/) (2021)
- [ProS: Facial Omni-Representation Learning via Prototype-based Self-Distillation](https://mlsanity.com/paper/2311.01929/) (2023)
- [Center Loss Regularization for Continual Learning](https://mlsanity.com/paper/2110.11314/) (2021)
- [OTFace: Hard Samples Guided Optimal Transport Loss for Deep Face Representation](https://mlsanity.com/paper/2203.14461/) (2022)
- [Small Temperature is All You Need for Differentiable Architecture Search](https://mlsanity.com/paper/2306.06855/) (2023)
- [Virtual Classification: Modulating Domain-Specific Knowledge for Multidomain Crowd Counting](https://mlsanity.com/paper/2402.03758/) (2024)
- [SymFace: Additional Facial Symmetry Loss for Deep Face Recognition](https://mlsanity.com/paper/2409.11816/) (2024)
- [GB-CosFace: Rethinking Softmax-based Face Recognition from the Perspective of Open Set Classification](https://mlsanity.com/paper/2111.11186/) (2021)
- [Perception Reinforcement Using Auxiliary Learning Feature Fusion: A Modified Yolov8 for Head Detection](https://mlsanity.com/paper/2310.09492/) (2023)
- [Age-Defying Face Recognition with Transformer-Enhanced Loss](https://mlsanity.com/paper/2412.02198/) (2024)
- [Jacobian-guided Noise Injection for Quantization Robustness in Large Language Models](https://mlsanity.com/paper/2608.20988/) (2026)
- [Re-calibrated Contrastive Loss for Transformation-Aware Prompt Conditioning in Vision-Language Models](https://mlsanity.com/paper/2609.06967/) (2026)

## Abstract (arXiv)

Over the past few years, softmax and SGD have become a commonly used component and the default training strategy in CNN frameworks, respectively. However, when optimizing CNNs with SGD, the saturation behavior behind softmax always gives us an illusion of training well and then is omitted. In this paper, we first emphasize that the early saturation behavior of softmax will impede the exploration of SGD, which sometimes is a reason for model converging at a bad local-minima, then propose Noisy Softmax to mitigating this early saturation issue by injecting annealed noise in softmax during each iteration. This operation based on noise injection aims at postponing the early saturation and further bringing continuous gradients propagation so as to significantly encourage SGD solver to be more exploratory and help to find a better local-minima. This paper empirically verifies the superiority of the early softmax desaturation, and our method indeed improves the generalization ability of CNN model by regularization. We experimentally find that this early desaturation helps optimization in many tasks, yielding state-of-the-art or competitive results on several popular benchmark datasets.