# Neural Anisotropy Directions

arXiv: https://arxiv.org/abs/2006.09717 · HTML: https://mlsanity.com/paper/2006.09717/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jun 2020.

## Built on by (most cited first, 17 of 17)

- [What can linearized neural networks actually say about generalization?](https://mlsanity.com/paper/2106.06770/) (2021)
- [Spectral Bias in Practice: The Role of Function Frequency in Generalization](https://mlsanity.com/paper/2110.02424/) (2021)
- [Spectral Evolution and Invariance in Linear-width Neural Networks](https://mlsanity.com/paper/2211.06506/) (2022)
- [On the Ability of Deep Networks to Learn Symmetries from Data: A Neural Kernel Theory](https://mlsanity.com/paper/2412.11521/) (2024)
- [A Directional Diffusion Graph Transformer for Recommendation](https://mlsanity.com/paper/2404.03326/) (2024)
- [A law of adversarial risk, interpolation, and label noise](https://mlsanity.com/paper/2207.03933/) (2022)
- [AA-SVD : Anchored and Adaptive SVD for Large Language Model Compression](https://mlsanity.com/paper/2604.02119/) (2026)
- [A neural anisotropic view of underspecification in deep learning](https://mlsanity.com/paper/2104.14372/) (2021)
- [What training reveals about neural network complexity](https://mlsanity.com/paper/2106.04186/) (2021)
- [Fairness and underspecification in acoustic scene classification: The case for disaggregated evaluations](https://mlsanity.com/paper/2110.01506/) (2021)
- [Understanding robustness and generalization of artificial neural networks through Fourier masks](https://mlsanity.com/paper/2203.08822/) (2022)
- [Catastrophic overfitting can be induced with discriminative non-robust features](https://mlsanity.com/paper/2206.08242/) (2022)
- [Geometric Inductive Biases of Deep Networks: The Role of Data and Architecture](https://mlsanity.com/paper/2410.12025/) (2024)
- [Towards Understanding the Power and Limits of the Muon Optimizer: A River-Valley Perspective](https://mlsanity.com/paper/2606.21514/) (2026)
- [On the Anisotropy of Score-Based Generative Models](https://mlsanity.com/paper/2510.22899/) (2025)
- [On Procedural Adversarial Noise Attack And Defense](https://mlsanity.com/paper/2108.04409/) (2021)
- [Topology-Enhanced Alignment for Large Language Models: Trajectory Topology Loss and Topological Preference Optimization](https://mlsanity.com/paper/2605.07172/) (2026)

## Abstract (arXiv)

In this work, we analyze the role of the network architecture in shaping the inductive bias of deep classifiers. To that end, we start by focusing on a very simple problem, i.e., classifying a class of linearly separable distributions, and show that, depending on the direction of the discriminative feature of the distribution, many state-of-the-art deep convolutional neural networks (CNNs) have a surprisingly hard time solving this simple task. We then define as neural anisotropy directions (NADs) the vectors that encapsulate the directional inductive bias of an architecture. These vectors, which are specific for each architecture and hence act as a signature, encode the preference of a network to separate the input data based on some particular features. We provide an efficient method to identify NADs for several CNN architectures and thus reveal their directional inductive biases. Furthermore, we show that, for the CIFAR-10 dataset, NADs characterize the features used by CNNs to discriminate between different classes.