# Geometry of Singular Foliations and Learning Manifolds in ReLU Networks via the Data Information Matrix

arXiv: https://arxiv.org/abs/2409.07412 · HTML: https://mlsanity.com/paper/2409.07412/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 46 references that are in the map.
- First posted on arXiv 11 Sep 2024.

## Builds on (most cited first, 17 of 17)

- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Automatic differentiation in machine learning: a survey](https://mlsanity.com/paper/1502.05767/) (2015)
- [New insights and perspectives on the natural gradient method](https://mlsanity.com/paper/1412.1193/) (2014)
- [Deep Learning for Classical Japanese Literature](https://mlsanity.com/paper/1812.01718/) (2018)
- [On the Number of Linear Regions of Deep Neural Networks](https://mlsanity.com/paper/1402.1869/) (2014)
- [EMNIST: an extension of MNIST to handwritten letters](https://mlsanity.com/paper/1702.05373/) (2017)
- [The Benefit of Multitask Representation Learning](https://mlsanity.com/paper/1505.06279/) (2015)
- [Complexity of Linear Regions in Deep Networks](https://mlsanity.com/paper/1901.09021/) (2019)
- [Deep ReLU Networks Have Surprisingly Few Activation Patterns](https://mlsanity.com/paper/1906.00904/) (2019)
- [Geometric Dataset Distances via Optimal Transport](https://mlsanity.com/paper/2002.02923/) (2020)
- [Dynamic Flows on Curved Space Generated by Labeled Data](https://mlsanity.com/paper/2302.00061/) (2023)
- [Model-centric Data Manifold: the Data Through the Eyes of the Model](https://mlsanity.com/paper/2104.13289/) (2021)
- [A Geometric Modeling of Occam's Razor in Deep Learning](https://mlsanity.com/paper/1905.11027/) (2019)
- [Data-driven reduced order models using invariant foliations, manifolds and autoencoders](https://mlsanity.com/paper/2206.12269/) (2022)
- [For Manifold Learning, Deep Neural Networks can be Locality Sensitive Hash Functions](https://mlsanity.com/paper/2103.06875/) (2021)
- [Adversarial attacks on neural networks through canonical Riemannian foliations](https://mlsanity.com/paper/2203.00922/) (2022)
- [Ricci Curvature and the Manifold Learning Problem](https://mlsanity.com/paper/1410.3351/) (2014)

## Built on by (most cited first, 1 of 1)

- [Cartan moving frames and the data manifolds](https://mlsanity.com/paper/2409.12057/) (2024)

## Abstract (arXiv)

Understanding how real data is distributed in high dimensional spaces is the key to many tasks in machine learning. We want to provide a natural geometric structure on the space of data employing a ReLU neural network trained as a classifier. Through the Data Information Matrix (DIM), a variation of the Fisher information matrix, the model will discern a singular foliation structure on the space of data. We show that the singular points of such foliation are contained in a measure zero set, and that a local regular foliation exists almost everywhere. Experiments show that the data is correlated with leaves of such foliation. Moreover we show the potential of our approach for knowledge transfer by analyzing the spectrum of the DIM to measure distances between datasets.