Gradient representations in ReLU networks as similarity functions
Not on the map. Why?
Abstract (arXiv)
Feed-forward networks can be interpreted as mappings with linear decision surfaces at the level of the last layer. We investigate how the tangent space of the network can be exploited to refine the decision in case of ReLU (Rectified Linear Unit) activations. We show that a simple Riemannian metric parametrized on the parameters of the network forms a similarity function at least as good as the original network and we suggest a sparse metric to increase the similarity gap.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 9 of its 18 references that are in the map. First posted on arXiv 26 Oct 2021.
Builds on
9 of 18 references.
References · 9
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
- 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
- 2016On the Expressive Power of Deep Neural Networkscited by 306
- 2018Sensitivity and Generalization in Neural Networks: an Empirical Studycited by 189
- 2019Complexity of Linear Regions in Deep Networkscited by 167
- 2020Empirical Studies on the Properties of Linear Regions in Deep Neural Networkscited by 36
- 2013Riemannian metrics for neural networks I: feedforward networkscited by 30
- 2021Deep ReLU Networks Preserve Expected Lengthcited by 17
- 2019Tangent Space Separability in Feedforward Neural Networkscited by 1
Built on by
Citing papers per year: 2023: 1.
1 paper.