# A singular Riemannian Geometry Approach to Deep Neural Networks III. Piecewise Differentiable Layers and Random Walks on n-dimensional Classes

arXiv: https://arxiv.org/abs/2404.06104 · HTML: https://mlsanity.com/paper/2404.06104/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 64 references that are in the map.
- First posted on arXiv 9 Apr 2024.

## Builds on (most cited first, 29 of 29)

- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Deep Image Prior](https://mlsanity.com/paper/1711.10925/) (2017)
- [Geometric deep learning: going beyond Euclidean data](https://mlsanity.com/paper/1611.08097/) (2016)
- [Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models](https://mlsanity.com/paper/2402.17177/) (2024)
- [Geometric deep learning on graphs and manifolds using mixture model CNNs](https://mlsanity.com/paper/1611.08402/) (2016)
- [On the Expressive Power of Deep Neural Networks](https://mlsanity.com/paper/1606.05336/) (2016)
- [Deep Convolutional Networks on Graph-Structured Data](https://mlsanity.com/paper/1506.05163/) (2015)
- [Riemannian Score-Based Generative Modelling](https://mlsanity.com/paper/2202.02763/) (2022)
- [Universal Approximation with Deep Narrow Networks](https://mlsanity.com/paper/1905.08539/) (2019)
- [Deep ReLU Networks Have Surprisingly Few Activation Patterns](https://mlsanity.com/paper/1906.00904/) (2019)
- [Learning shape correspondence with anisotropic convolutional neural networks](https://mlsanity.com/paper/1605.06437/) (2016)
- [Reverse-Engineering Deep ReLU Networks](https://mlsanity.com/paper/1910.00744/) (2019)
- [Diffusion Models for Constrained Domains](https://mlsanity.com/paper/2304.05364/) (2023)
- [The Geometry of Deep Networks: Power Diagram Subdivision](https://mlsanity.com/paper/1905.08443/) (2019)
- [Interpreting Neural Networks through the Polytope Lens](https://mlsanity.com/paper/2211.12312/) (2022)
- [Community detection in node-attributed social networks: a survey](https://mlsanity.com/paper/1912.09816/) (2019)
- [Combining Weighted Total Variation and Deep Image Prior for natural and medical image restoration via ADMM](https://mlsanity.com/paper/2009.11380/) (2020)
- [Constructing Long Short-Term Memory based Deep Recurrent Neural Networks for Large Vocabulary Speech Recognition](https://mlsanity.com/paper/1410.4281/) (2014)
- [ReLU Deep Neural Networks from the Hierarchical Basis Perspective](https://mlsanity.com/paper/2105.04156/) (2021)
- [A singular Riemannian geometry approach to Deep Neural Networks I. Theoretical foundations](https://mlsanity.com/paper/2201.09656/) (2021)
- [Origami in N dimensions: How feed-forward networks manufacture linear separability](https://mlsanity.com/paper/2203.11355/) (2022)
- [A singular Riemannian geometry approach to Deep Neural Networks II. Reconstruction of 1-D equivalence classes](https://mlsanity.com/paper/2112.10583/) (2021)
- [Intriguing Equivalence Structures of the Embedding Space of Vision Transformers](https://mlsanity.com/paper/2401.15568/) (2024)
- [A Differential Topological View of Challenges in Learning with Feedforward Neural Networks](https://mlsanity.com/paper/1811.10304/) (2018)
- [BERT-DRE: BERT with Deep Recursive Encoder for Natural Language Sentence Matching](https://mlsanity.com/paper/2111.02188/) (2021)
- [Probabilistic Risk Assessment of an Obstacle Detection System for GoA 4 Freight Trains](https://mlsanity.com/paper/2306.14814/) (2023)
- [A Stochastic Approach to Classification Error Estimates in Convolutional Neural Networks](https://mlsanity.com/paper/2401.06156/) (2023)
- [Exploring Geometry of Blind Spots in Vision Models](https://mlsanity.com/paper/2310.19889/) (2023)

## Built on by (most cited first, 2 of 2)

- [Unveiling Transformer Perception by Exploring Input Manifolds](https://mlsanity.com/paper/2410.06019/) (2024)
- [Riemannian Optimization in Modular Systems](https://mlsanity.com/paper/2603.03610/) (2026)

## Abstract (arXiv)

Neural networks are playing a crucial role in everyday life, with the most modern generative models able to achieve impressive results. Nonetheless, their functioning is still not very clear, and several strategies have been adopted to study how and why these model reach their outputs. A common approach is to consider the data in an Euclidean settings: recent years has witnessed instead a shift from this paradigm, moving thus to more general framework, namely Riemannian Geometry. Two recent works introduced a geometric framework to study neural networks making use of singular Riemannian metrics. In this paper we extend these results to convolutional, residual and recursive neural networks, studying also the case of non-differentiable activation functions, such as ReLU. We illustrate our findings with some numerical experiments on classification of images and thermodynamic problems.