# On Geometric Structure of Activation Spaces in Neural Networks

arXiv: https://arxiv.org/abs/1904.01399 · HTML: https://mlsanity.com/paper/1904.01399/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Apr 2019.

## Built on by (most cited first, 4 of 4)

- [Hyper-Representations: Self-Supervised Representation Learning on Neural Network Weights for Model Characteristic Prediction](https://mlsanity.com/paper/2110.15288/) (2021)
- [Solving hybrid machine learning tasks by traversing weight space geodesics](https://mlsanity.com/paper/2106.02793/) (2021)
- [Why Robust Natural Language Understanding is a Challenge](https://mlsanity.com/paper/2206.14575/) (2022)
- [Hyper-Representations: Learning from Populations of Neural Networks](https://mlsanity.com/paper/2410.05107/) (2024)

## Abstract (arXiv)

In this paper, we investigate the geometric structure of activation spaces of fully connected layers in neural networks and then show applications of this study. We propose an efficient approximation algorithm to characterize the convex hull of massive points in high dimensional space. Based on this new algorithm, four common geometric properties shared by the activation spaces are concluded, which gives a rather clear description of the activation spaces. We then propose an alternative classification method grounding on the geometric structure description, which works better than neural networks alone. Surprisingly, this data classification method can be an indicator of overfitting in neural networks. We believe our work reveals several critical intrinsic properties of modern neural networks and further gives a new metric for evaluating them.