# Tangent Space Separability in Feedforward Neural Networks

arXiv: https://arxiv.org/abs/1912.09306 · HTML: https://mlsanity.com/paper/1912.09306/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Dec 2019.

## Built on by (most cited first, 1 of 1)

- [Gradient representations in ReLU networks as similarity functions](https://mlsanity.com/paper/2110.13581/) (2021)

## Abstract (arXiv)

Hierarchical neural networks are exponentially more efficient than their corresponding "shallow" counterpart with the same expressive power, but involve huge number of parameters and require tedious amounts of training. By approximating the tangent subspace, we suggest a sparse representation that enables switching to shallow networks, GradNet after a very early training stage. Our experiments show that the proposed approximation of the metric improves and sometimes even surpasses the achievable performance of the original network significantly even after a few epochs of training the original feedforward network.