# Reframing Neural Networks: Deep Structure in Overcomplete Representations

arXiv: https://arxiv.org/abs/2103.05804 · HTML: https://mlsanity.com/paper/2103.05804/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 67 references that are in the map.
- First posted on arXiv 10 Mar 2021.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [Train faster, generalize better: Stability of stochastic gradient descent](https://mlsanity.com/paper/1509.01240/) (2015)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [Parseval Networks: Improving Robustness to Adversarial Examples](https://mlsanity.com/paper/1704.08847/) (2017)
- [Predicting Parameters in Deep Learning](https://mlsanity.com/paper/1306.0543/) (2013)
- [Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks](https://mlsanity.com/paper/1905.13210/) (2019)
- [Sensitivity and Generalization in Neural Networks: an Empirical Study](https://mlsanity.com/paper/1802.08760/) (2018)
- [Learning the Number of Neurons in Deep Networks](https://mlsanity.com/paper/1611.06321/) (2016)
- [Convolutional Neural Networks Analyzed via Convolutional Sparse Coding](https://mlsanity.com/paper/1607.08194/) (2016)
- [Deep Convolutional Compressed Sensing for LiDAR Depth Completion](https://mlsanity.com/paper/1803.08949/) (2018)
- [Deep Neural Network Structures Solving Variational Inequalities](https://mlsanity.com/paper/1808.07526/) (2018)
- [On Multi-Layer Basis Pursuit, Efficient Algorithms and Convolutional Neural Networks](https://mlsanity.com/paper/1806.00701/) (2018)
- [Architectural Adversarial Robustness: The Case for Deep Pursuit](https://mlsanity.com/paper/2011.14427/) (2020)
- [Adversarial Noise Attacks of Deep Learning Architectures -- Stability Analysis via Sparse Modeled Signals](https://mlsanity.com/paper/1805.11596/) (2018)
- [Dataless Model Selection with the Deep Frame Potential](https://mlsanity.com/paper/2003.13866/) (2020)
- [Dimensionality-Dependent Generalization Bounds for k-Dimensional Coding Schemes](https://mlsanity.com/paper/1601.00238/) (2016)
- [Learning overcomplete, low coherence dictionaries with linear inference](https://mlsanity.com/paper/1606.03474/) (2016)
- [Deep Component Analysis via Alternating Direction Neural Networks](https://mlsanity.com/paper/1803.06407/) (2018)

## Built on by (most cited first, 1 of 1)

- [Structural Extensions of Basis Pursuit: Guarantees on Adversarial Robustness](https://mlsanity.com/paper/2205.08955/) (2022)

## Abstract (arXiv)

In comparison to classical shallow representation learning techniques, deep neural networks have achieved superior performance in nearly every application benchmark. But despite their clear empirical advantages, it is still not well understood what makes them so effective. To approach this question, we introduce deep frame approximation: a unifying framework for constrained representation learning with structured overcomplete frames. While exact inference requires iterative optimization, it may be approximated by the operations of a feed-forward deep neural network. We indirectly analyze how model capacity relates to frame structures induced by architectural hyperparameters such as depth, width, and skip connections. We quantify these structural differences with the deep frame potential, a data-independent measure of coherence linked to representation uniqueness and stability. As a criterion for model selection, we show correlation with generalization error on a variety of common deep network architectures and datasets. We also demonstrate how recurrent networks implementing iterative optimization algorithms can achieve performance comparable to their feed-forward approximations while improving adversarial robustness. This connection to the established theory of overcomplete representations suggests promising new directions for principled deep network architecture design with less reliance on ad-hoc engineering.