# Gradient representations in ReLU networks as similarity functions

arXiv: https://arxiv.org/abs/2110.13581 · HTML: https://mlsanity.com/paper/2110.13581/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 18 references that are in the map.
- First posted on arXiv 26 Oct 2021.

## Builds on (most cited first, 9 of 9)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [On the Expressive Power of Deep Neural Networks](https://mlsanity.com/paper/1606.05336/) (2016)
- [Sensitivity and Generalization in Neural Networks: an Empirical Study](https://mlsanity.com/paper/1802.08760/) (2018)
- [Complexity of Linear Regions in Deep Networks](https://mlsanity.com/paper/1901.09021/) (2019)
- [Empirical Studies on the Properties of Linear Regions in Deep Neural Networks](https://mlsanity.com/paper/2001.01072/) (2020)
- [Riemannian metrics for neural networks I: feedforward networks](https://mlsanity.com/paper/1303.0818/) (2013)
- [Deep ReLU Networks Preserve Expected Length](https://mlsanity.com/paper/2102.10492/) (2021)
- [Tangent Space Separability in Feedforward Neural Networks](https://mlsanity.com/paper/1912.09306/) (2019)

## Built on by (most cited first, 1 of 1)

- [Optimization dependent generalization bound for ReLU networks based on sensitivity in the tangent bundle](https://mlsanity.com/paper/2310.17378/) (2023)

## Abstract (arXiv)

Feed-forward networks can be interpreted as mappings with linear decision surfaces at the level of the last layer. We investigate how the tangent space of the network can be exploited to refine the decision in case of ReLU (Rectified Linear Unit) activations. We show that a simple Riemannian metric parametrized on the parameters of the network forms a similarity function at least as good as the original network and we suggest a sparse metric to increase the similarity gap.