# Neural Network Layer Algebra: A Framework to Measure Capacity and Compression in Deep Learning

arXiv: https://arxiv.org/abs/2107.01081 · HTML: https://mlsanity.com/paper/2107.01081/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 57 references that are in the map.
- First posted on arXiv 2 Jul 2021.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Fantastic Generalization Measures and Where to Find Them](https://mlsanity.com/paper/1912.02178/) (2019)
- [On the Number of Linear Regions of Deep Neural Networks](https://mlsanity.com/paper/1402.1869/) (2014)
- [How Does Batch Normalization Help Optimization?](https://mlsanity.com/paper/1805.11604/) (2018)
- [The Expressive Power of Neural Networks: A View from the Width](https://mlsanity.com/paper/1709.02540/) (2017)
- [Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks](https://mlsanity.com/paper/1703.02930/) (2017)
- [On the Expressive Power of Deep Neural Networks](https://mlsanity.com/paper/1606.05336/) (2016)
- [Sensitivity and Generalization in Neural Networks: an Empirical Study](https://mlsanity.com/paper/1802.08760/) (2018)
- [On the Properties of the Softmax Function with Application in Game Theory and Reinforcement Learning](https://mlsanity.com/paper/1704.00805/) (2017)
- [An Analysis of Deep Neural Network Models for Practical Applications](https://mlsanity.com/paper/1605.07678/) (2016)
- [Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activations](https://mlsanity.com/paper/1708.02691/) (2017)
- [On the Expressive Power of Deep Learning: A Tensor Analysis](https://mlsanity.com/paper/1509.05009/) (2015)
- [On the number of response regions of deep feed forward networks with piece-wise linear activations](https://mlsanity.com/paper/1312.6098/) (2013)
- [Neural Persistence: A Complexity Measure for Deep Neural Networks Using Algebraic Topology](https://mlsanity.com/paper/1812.09764/) (2018)
- [A Mean-Field Optimal Control Formulation of Deep Learning](https://mlsanity.com/paper/1807.01083/) (2018)
- [The power of deeper networks for expressing natural functions](https://mlsanity.com/paper/1705.05502/) (2017)
- [Approximation and Estimation for High-Dimensional Deep Learning Networks](https://mlsanity.com/paper/1809.03090/) (2018)
- [Learning Functions: When Is Deep Better Than Shallow](https://mlsanity.com/paper/1603.00988/) (2016)
- [On the Complexity of Learning Neural Networks](https://mlsanity.com/paper/1707.04615/) (2017)
- [Architectural Complexity Measures of Recurrent Neural Networks](https://mlsanity.com/paper/1602.08210/) (2016)
- [Complexity, Statistical Risk, and Metric Entropy of Deep Nets Using Total Path Variation](https://mlsanity.com/paper/1902.00800/) (2019)
- [Stochastic Training of Residual Networks: a Differential Equation Viewpoint](https://mlsanity.com/paper/1812.00174/) (2018)
- [Mathematics of Deep Learning](https://mlsanity.com/paper/1712.04741/) (2017)
- [Size and Depth Separation in Approximating Benign Functions with Neural Networks](https://mlsanity.com/paper/2102.00314/) (2021)
- [Generating Albums with SampleRNN to Imitate Metal, Rock, and Punk Bands](https://mlsanity.com/paper/1811.06633/) (2018)

## Built on by (most cited first, 2 of 2)

- [A Distance Correlation-Based Approach to Characterize the Effectiveness of Recurrent Neural Networks for Time Series Forecasting](https://mlsanity.com/paper/2307.15830/) (2023)
- [Graph Lineages and Skeletal Graph Products](https://mlsanity.com/paper/2508.00197/) (2025)

## Abstract (arXiv)

We present a new framework to measure the intrinsic properties of (deep) neural networks. While we focus on convolutional networks, our framework can be extrapolated to any network architecture. In particular, we evaluate two network properties, namely, capacity, which is related to expressivity, and compression, which is related to learnability. Both these properties depend only on the network structure and are independent of the network parameters. To this end, we propose two metrics: the first one, called layer complexity, captures the architectural complexity of any network layer; and, the second one, called layer intrinsic power, encodes how data is compressed along the network. The metrics are based on the concept of layer algebra, which is also introduced in this paper. This concept is based on the idea that the global properties depend on the network topology, and the leaf nodes of any neural network can be approximated using local transfer functions, thereby, allowing a simple computation of the global metrics. We show that our global complexity metric can be calculated and represented more conveniently than the widely-used VC dimension. We also compare the properties of various state-of-the art architectures using our metrics and use the properties to analyze their accuracy on benchmark image classification datasets.