# Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks

arXiv: https://arxiv.org/abs/1702.05870 · HTML: https://mlsanity.com/paper/1702.05870/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Feb 2017.

## Built on by (most cited first, 30 of 59)

- [Analyzing and Improving the Training Dynamics of Diffusion Models](https://mlsanity.com/paper/2312.02696/) (2023)
- [B-cos Networks: Alignment is All We Need for Interpretability](https://mlsanity.com/paper/2205.10268/) (2022)
- [Goal-Oriented Gaze Estimation for Zero-Shot Learning](https://mlsanity.com/paper/2103.03433/) (2021)
- [Essentials for Class Incremental Learning](https://mlsanity.com/paper/2102.09517/) (2021)
- [Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length](https://mlsanity.com/paper/2404.08801/) (2024)
- [B-cos Alignment for Inherently Interpretable CNNs and Vision Transformers](https://mlsanity.com/paper/2306.10898/) (2023)
- [Class-Balanced Distillation for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2104.05279/) (2021)
- [An EM Framework for Online Incremental Learning of Semantic Segmentation](https://mlsanity.com/paper/2108.03613/) (2021)
- [Task-Aware Information Routing from Common Representation Space in Lifelong Learning](https://mlsanity.com/paper/2302.11346/) (2023)
- [Continual Learning for Visual Search with Backward Consistent Feature Embedding](https://mlsanity.com/paper/2205.13384/) (2022)
- [A Unified Framework for Generalized Low-Shot Medical Image Segmentation with Scarce Data](https://mlsanity.com/paper/2110.09260/) (2021)
- [Tackling Catastrophic Forgetting and Background Shift in Continual Semantic Segmentation](https://mlsanity.com/paper/2106.15287/) (2021)
- [Detecting Anomalies in Semantic Segmentation with Prototypes](https://mlsanity.com/paper/2106.00472/) (2021)
- [Continual Causal Inference with Incremental Observational Data](https://mlsanity.com/paper/2303.01775/) (2023)
- [Large-scale Pre-trained Models are Surprisingly Strong in Incremental Novel Class Discovery](https://mlsanity.com/paper/2303.15975/) (2023)
- [D3Former: Debiased Dual Distilled Transformer for Incremental Learning](https://mlsanity.com/paper/2208.00777/) (2022)
- [Single-Stage Heavy-Tailed Food Classification](https://mlsanity.com/paper/2307.00182/) (2023)
- [Feature Generation for Long-tail Classification](https://mlsanity.com/paper/2111.05956/) (2021)
- [Surpassing Cosine Similarity for Multidimensional Comparisons: Dimension Insensitive Euclidean Metric](https://mlsanity.com/paper/2407.08623/) (2024)
- [Connecting Giants: Synergistic Knowledge Transfer of Large Multimodal Models for Few-Shot Learning](https://mlsanity.com/paper/2510.11115/) (2025)
- [SHELS: Exclusive Feature Sets for Novelty Detection and Continual Learning Without Class Boundaries](https://mlsanity.com/paper/2206.13720/) (2022)
- [Swinv2-Imagen: Hierarchical Vision Transformer Diffusion Models for Text-to-Image Generation](https://mlsanity.com/paper/2210.09549/) (2022)
- [YONA: You Only Need One Adjacent Reference-frame for Accurate and Fast Video Polyp Detection](https://mlsanity.com/paper/2306.03686/) (2023)
- [GPS: A Policy-driven Sampling Approach for Graph Representation Learning](https://mlsanity.com/paper/2112.14482/) (2021)
- [Cottention: Linear Transformers With Cosine Attention](https://mlsanity.com/paper/2409.18747/) (2024)
- [Rethinking Layer-wise Model Merging through Chain of Merges](https://mlsanity.com/paper/2508.21421/) (2025)
- [Enhancing Few-shot Image Classification with Cosine Transformer](https://mlsanity.com/paper/2211.06828/) (2022)
- [A Universal Metric of Dataset Similarity for Cross-silo Federated Learning](https://mlsanity.com/paper/2404.18773/) (2024)
- [Enabling Fast Deep Learning on Tiny Energy-Harvesting IoT Devices](https://mlsanity.com/paper/2111.14051/) (2021)
- [Cross-Shaped Windows Transformer with Self-supervised Pretraining for Clinically Significant Prostate Cancer Detection in Bi-parametric MRI](https://mlsanity.com/paper/2305.00385/) (2023)

## Abstract (arXiv)

Traditionally, multi-layer neural networks use dot product between the output vector of previous layer and the incoming weight vector as the input to activation function. The result of dot product is unbounded, thus increases the risk of large variance. Large variance of neuron makes the model sensitive to the change of input distribution, thus results in poor generalization, and aggravates the internal covariate shift which slows down the training. To bound dot product and decrease the variance, we propose to use cosine similarity or centered cosine similarity (Pearson Correlation Coefficient) instead of dot product in neural networks, which we call cosine normalization. We compare cosine normalization with batch, weight and layer normalization in fully-connected neural networks as well as convolutional networks on the data sets of MNIST, 20NEWS GROUP, CIFAR-10/100 and SVHN. Experiments show that cosine normalization achieves better performance than other normalization techniques.