# A Geometric Modeling of Occam's Razor in Deep Learning

arXiv: https://arxiv.org/abs/1905.11027 · HTML: https://mlsanity.com/paper/1905.11027/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 May 2019.

## Built on by (most cited first, 8 of 8)

- [Trade-Offs of Diagonal Fisher Information Matrix Estimators](https://mlsanity.com/paper/2402.05379/) (2024)
- [GeLoRA: Geometric Adaptive Ranks For Efficient LoRA Fine-tuning](https://mlsanity.com/paper/2412.09250/) (2024)
- [Soft ascent-descent as a stable and flexible alternative to flooding](https://mlsanity.com/paper/2310.10006/) (2023)
- [Information-Theoretic Framework for Understanding Modern Machine-Learning](https://mlsanity.com/paper/2506.07661/) (2025)
- [Geometry of Singular Foliations and Learning Manifolds in ReLU Networks via the Data Information Matrix](https://mlsanity.com/paper/2409.07412/) (2024)
- [Normalized Maximum Likelihood Code-Length on Riemannian Data Spaces](https://mlsanity.com/paper/2508.21466/) (2025)
- [Deterministic Bounds and Random Estimates of Metric Tensors on Neuromanifolds](https://mlsanity.com/paper/2505.13614/) (2025)
- [Anchoring the Eigengap: Cross-Modal Spectral Stabilization for Sample-Efficient Representation Learning](https://mlsanity.com/paper/2605.08764/) (2026)

## Abstract (arXiv)

Why do deep neural networks (DNNs) benefit from very high dimensional parameter spaces? Their huge parameter complexities vs stunning performance in practice is all the more intriguing and not explainable using the standard theory of model selection for regular models. In this work, we propose a geometrically flavored information-theoretic approach to study this phenomenon. With the belief that simplicity is linked to better generalization, as grounded in the theory of minimum description length, the objective of our analysis is to examine and bound the complexity of DNNs. We introduce the locally varying dimensionality of the parameter space of neural network models by considering the number of significant dimensions of the Fisher information matrix, and model the parameter space as a manifold using the framework of singular semi-Riemannian geometry. We derive model complexity measures which yield short description lengths for deep neural network models based on their singularity analysis thus explaining the good performance of DNNs despite their large number of parameters.