# Understanding the Feature Norm for Out-of-Distribution Detection

arXiv: https://arxiv.org/abs/2310.05316 · HTML: https://mlsanity.com/paper/2310.05316/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 55 references that are in the map.
- First posted on arXiv 9 Oct 2023.

## Builds on (most cited first, 30 of 42)

- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [Generalized Out-of-Distribution Detection: A Survey](https://mlsanity.com/paper/2110.11334/) (2021)
- [Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images](https://mlsanity.com/paper/1412.1897/) (2014)
- [Out-of-Distribution Detection with Deep Nearest Neighbors](https://mlsanity.com/paper/2204.06507/) (2022)
- [Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery](https://mlsanity.com/paper/1703.05921/) (2017)
- [Empirical Evaluation of Rectified Activations in Convolutional Network](https://mlsanity.com/paper/1505.00853/) (2015)
- [Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1906.12340/) (2019)
- [ReAct: Out-of-distribution Detection With Rectified Activations](https://mlsanity.com/paper/2111.12797/) (2021)
- [Scaling Out-of-Distribution Detection for Real-World Settings](https://mlsanity.com/paper/1911.11132/) (2019)
- [Do Deep Generative Models Know What They Don't Know?](https://mlsanity.com/paper/1810.09136/) (2018)
- [CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances](https://mlsanity.com/paper/2007.08176/) (2020)
- [Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data](https://mlsanity.com/paper/2002.11297/) (2020)
- [Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One](https://mlsanity.com/paper/1912.03263/) (2019)
- [ViM: Out-Of-Distribution with Virtual-logit Matching](https://mlsanity.com/paper/2203.10807/) (2022)
- [Open-Set Recognition: a Good Closed-Set Classifier is All You Need?](https://mlsanity.com/paper/2110.06207/) (2021)
- [On the Importance of Gradients for Detecting Distributional Shifts in the Wild](https://mlsanity.com/paper/2110.00218/) (2021)
- [MagFace: A Universal Representation for Face Recognition and Quality Assessment](https://mlsanity.com/paper/2103.06627/) (2021)
- [Deep Anomaly Detection Using Geometric Transformations](https://mlsanity.com/paper/1805.10917/) (2018)
- [OpenOOD: Benchmarking Generalized Out-of-Distribution Detection](https://mlsanity.com/paper/2210.07242/) (2022)
- [SSD: A Unified Framework for Self-Supervised Outlier Detection](https://mlsanity.com/paper/2103.12051/) (2021)
- [MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Space](https://mlsanity.com/paper/2105.01879/) (2021)

## Built on by (most cited first, 27 of 27)

- [Detecting Out-of-Distribution Through the Lens of Neural Collapse](https://mlsanity.com/paper/2311.01479/) (2023)
- [Mahalanobis++: Improving OOD Detection via Feature Normalization](https://mlsanity.com/paper/2505.18032/) (2025)
- [Test-Time Discovery via Hashing Memory](https://mlsanity.com/paper/2503.10699/) (2025)
- [Enhancing Out-of-Distribution Detection with Extended Logit Normalization](https://mlsanity.com/paper/2504.11434/) (2025)
- [Hidden Conflicts in Neural Networks and Their Implications for Explainability](https://mlsanity.com/paper/2310.20363/) (2023)
- [One-Point Contraction: Erasing Representational Separability toward Irreversible Deep Forgetting](https://mlsanity.com/paper/2507.07754/) (2025)
- [RankFeat&RankWeight: Rank-1 Feature/Weight Removal for Out-of-distribution Detection](https://mlsanity.com/paper/2311.13959/) (2023)
- [Representation Norm Amplification for Out-of-Distribution Detection in Long-Tail Learning](https://mlsanity.com/paper/2408.10676/) (2024)
- [SphOR: A Representation Learning Perspective on Open-set Recognition for Identifying Unknown Classes in Deep Learning Models](https://mlsanity.com/paper/2503.08049/) (2025)
- [SEQR: Secure and Efficient QR-based LoRA Routing](https://mlsanity.com/paper/2509.18093/) (2025)
- [BootOOD: Self-Supervised Out-of-Distribution Detection via Synthetic Sample Exposure under Neural Collapse](https://mlsanity.com/paper/2511.13539/) (2025)
- [Mitigating the ID-OOD Tradeoff in Open-Set Test-Time Adaptation](https://mlsanity.com/paper/2604.01589/) (2026)
- [HyCal: A Training-Free Prototype Calibration Method for Cross-Discipline Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2604.15678/) (2026)
- [Direct Product Flow Matching: Decoupling Radial and Angular Dynamics for Few-Shot Adaptation](https://mlsanity.com/paper/2605.05054/) (2026)
- [FedLAS: Feature-Modulated Bidirectional Label Smoothing for Neural Network Calibration](https://mlsanity.com/paper/2606.28654/) (2026)
- [The Over-Certainty Phenomenon in Modern Test-Time Adaptation Algorithms](https://mlsanity.com/paper/2404.16168/) (2024)
- [Prompt Optimization Meets Subspace Representation Learning for Few-shot Out-of-Distribution Detection](https://mlsanity.com/paper/2509.18111/) (2025)
- [General OOD Detection via Model-aware and Subspace-aware Variable Priority](https://mlsanity.com/paper/2512.13003/) (2025)
- [The Invisible Gorilla Effect in Out-of-distribution Detection](https://mlsanity.com/paper/2602.20068/) (2026)
- [HEART-PFL: Stable Personalized Federated Learning under Heterogeneity with Hierarchical Directional Alignment and Adversarial Knowledge Transfer](https://mlsanity.com/paper/2603.24209/) (2026)
- [Knowledge-Guided Failure Prediction: Detecting When Object Detectors Miss Safety-Critical Objects](https://mlsanity.com/paper/2603.25499/) (2026)
- [Beyond Known Objects: A Novel Framework for Open-Set Object Detection using Negative-Aware Norm](https://mlsanity.com/paper/2605.02284/) (2026)
- [PoLAR: Factorizing Extent and Mode in Latent Actions for Robot Policy Learning](https://mlsanity.com/paper/2606.21139/) (2026)
- [Exploiting Local Flatness for Efficient Out-of-Distribution Detection](https://mlsanity.com/paper/2606.29952/) (2026)
- [Suppress and Diversify: Refining Robust Pathways for Corruption Robustness](https://mlsanity.com/paper/2608.06712/) (2026)
- [Energy Ring: Stabilizing Feature Energy for Reliable Test-Time Discovery](https://mlsanity.com/paper/2610.14525/) (2026)
- [Relational Semantic Grounding for Test-Time Discovery](https://mlsanity.com/paper/2610.14721/) (2026)

## Abstract (arXiv)

A neural network trained on a classification dataset often exhibits a higher vector norm of hidden layer features for in-distribution (ID) samples, while producing relatively lower norm values on unseen instances from out-of-distribution (OOD). Despite this intriguing phenomenon being utilized in many applications, the underlying cause has not been thoroughly investigated. In this study, we demystify this very phenomenon by scrutinizing the discriminative structures concealed in the intermediate layers of a neural network. Our analysis leads to the following discoveries: (1) The feature norm is a confidence value of a classifier hidden in the network layer, specifically its maximum logit. Hence, the feature norm distinguishes OOD from ID in the same manner that a classifier confidence does. (2) The feature norm is class-agnostic, thus it can detect OOD samples across diverse discriminative models. (3) The conventional feature norm fails to capture the deactivation tendency of hidden layer neurons, which may lead to misidentification of ID samples as OOD instances. To resolve this drawback, we propose a novel negative-aware norm (NAN) that can capture both the activation and deactivation tendencies of hidden layer neurons. We conduct extensive experiments on NAN, demonstrating its efficacy and compatibility with existing OOD detectors, as well as its capability in label-free environments.