# Pointwise Generalization in Deep Neural Networks

arXiv: https://arxiv.org/abs/2605.18598 · HTML: https://mlsanity.com/paper/2605.18598/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 132 references that are in the map.
- First posted on arXiv 18 May 2026.

## Builds on (most cited first, 12 of 12)

- [Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data](https://mlsanity.com/paper/1703.11008/) (2017)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [Uniform convergence may be unable to explain generalization in deep learning](https://mlsanity.com/paper/1902.04742/) (2019)
- [Deep Learning is Not So Mysterious or Different](https://mlsanity.com/paper/2503.02113/) (2025)
- [Understanding Deep Representation Learning via Layerwise Feature Compression and Discrimination](https://mlsanity.com/paper/2311.02960/) (2023)
- [Neural Tangent Kernel: A Survey](https://mlsanity.com/paper/2208.13614/) (2022)
- [Neural Collapse in the Intermediate Hidden Layers of Classification Neural Networks](https://mlsanity.com/paper/2308.02760/) (2023)
- [Six Lectures on Linearized Neural Networks](https://mlsanity.com/paper/2308.13431/) (2023)
- [An Overview of Low-Rank Structures in the Training and Adaptation of Large Models](https://mlsanity.com/paper/2503.19859/) (2025)
- [Learning to Compress: Local Rank and Information Compression in Deep Neural Networks](https://mlsanity.com/paper/2410.07687/) (2024)
- [Generalization Bounds for Rank-sparse Neural Networks](https://mlsanity.com/paper/2510.21945/) (2025)
- [The Two Lives of the Grassmannian](https://mlsanity.com/paper/2401.03684/) (2024)

## Built on by (most cited first, 4 of 4)

- [Pointwise Complexity for Gaussian Fields: Upper Envelopes, Algorithmic Lower Bounds, and Separation](https://mlsanity.com/paper/2606.07931/) (2026)
- [Bellman-sufficient Information Complexity](https://mlsanity.com/paper/2606.11171/) (2026)
- [Online Control via Counterfactual Tracking](https://mlsanity.com/paper/2607.13029/) (2026)
- [Local Search with Correlated Randomness](https://mlsanity.com/paper/2607.17469/) (2026)

## Abstract (arXiv)

We address the fundamental question of why deep neural networks generalize by establishing a pointwise generalization theory for fully connected networks. This framework resolves long-standing barriers to characterizing the rich nonlinear feature-learning regime and builds a new statistical foundation for representation learning. For each trained model, we characterize the hypothesis via a pointwise Riemannian Dimension, derived from the eigenvalues of the learned feature representations across layers. This establishes a principled framework for deriving hypothesis-dependent, representation-aware generalization bounds. These bounds offer a systematic upgrade over approaches based on model size, products of norms, and infinite-width linearizations, yielding guarantees that are orders of magnitude tighter in both theory and experiment. Analytically, we identify the structural properties and mathematical principles that explain the tractability of deep networks. Empirically, the pointwise Riemannian Dimension exhibits substantial feature compression, decreases with increased over-parameterization, and captures the implicit bias of optimizers. Taken together, our results indicate that deep networks are mathematically tractable in practical regimes and that their generalization is sharply explained by pointwise, feature-spectrum-aware complexity.