Towards Theoretical Analysis of Transformation Complexity of ReLU DNNs
Not on the map. Why?
Abstract (arXiv)
This paper aims to theoretically analyze the complexity of feature transformations encoded in piecewise linear DNNs with ReLU layers. We propose metrics to measure three types of complexities of transformations based on the information theory. We further discover and prove the strong correlation between the complexity and the disentanglement of transformations. Based on the proposed metrics, we analyze two typical phenomena of the change of the transformation complexity during the training process, and explore the ceiling of a DNN's complexity. The proposed metrics can also be used as a loss to learn a DNN with the minimum complexity, which also controls the over-fitting level of the DNN and influences adversarial robustness, adversarial transferability, and knowledge consistency. Comprehensive comparative studies have provided new perspectives to understand the DNN.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 49 of its 64 references that are in the map. First posted on arXiv 4 May 2022.
Builds on
49 of 64 references.
References · 49
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
- 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
- 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
- 2013Auto-Encoding Variational Bayescited by 3,980
- 2014Deep Learning Face Attributes in the Wildcited by 3,679
- 2015Learning Deep Features for Discriminative Localizationcited by 2,430
- 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
- 2013Visualizing and Understanding Convolutional Networkscited by 2,278
- 2016Identity Mappings in Deep Residual Networkscited by 1,960
37 more
- 2019Similarity of Neural Network Representations Revisitedcited by 1,695
- 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
- 2017Interpretable Explanations of Black Boxes by Meaningful Perturbationcited by 611
- 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
- 2014Understanding Deep Image Representations by Inverting Themcited by 399
- 2015Understanding Neural Networks Through Deep Visualizationcited by 332
- 2018Understanding disentangling in β-VAEcited by 332
- 2016On the Expressive Power of Deep Neural Networkscited by 302
- 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 299
- 2017Three Factors Influencing Minima in SGDcited by 270
- 2016Understanding Deep Neural Networks with Rectified Linear Unitscited by 253
- 2018Learning to Explain: An Information-Theoretic Perspective on Model Interpretationcited by 248
- 2017Emergence of Invariance and Disentanglement in Deep Representationscited by 213
- 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 203
- 2018Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approachcited by 181
- 2014Object Detectors Emerge in Deep Scene CNNscited by 178
- 2015Inverting Visual Representations with Convolutional Networkscited by 168
- 2016Information Dropout: Learning Optimal Representations Through Noisy Computationcited by 131
- 2020Learning Latent Space Energy-Based Prior Modelcited by 104
- 2017Fisher-Rao Metric, Geometry, and Complexity of Neural Networkscited by 99
- 2017Learning how to explain neural networks: PatternNet and PatternAttributioncited by 84
- 2014On the Computational Efficiency of Training Neural Networkscited by 71
- 2017Nonlinear Information Bottleneckcited by 71
- 2020A Unified Approach to Interpreting and Boosting Adversarial Transferabilitycited by 71
- 2013How to Construct Deep Recurrent Neural Networkscited by 64
- 2016AdaNet: Adaptive Structural Learning of Artificial Neural Networkscited by 54
- 2019Stiffness: A New Perspective on Generalization in Neural Networkscited by 46
- 2018Understanding training and generalization in deep learning by Fourier analysiscited by 43
- 2020Early Stopping in Deep Networks: Double Descent and How to Eliminate itcited by 40
- 2017Estimating Mixture Entropy with Pairwise Distancescited by 34
- 2017Learning Energy-Based Models as Generative ConvNets via Multi-grid Modeling and Samplingcited by 32
- 2018Complexity of Training ReLU Neural Networkcited by 23
- 2018Scalable Mutual Information Estimation using Dependence Graphscited by 22
- 2016Architectural Complexity Measures of Recurrent Neural Networkscited by 17
- 2019Knowledge Consistency between Neural Networks and Beyondcited by 16
- 2014Maximally Informative Hierarchical Representations of High-Dimensional Datacited by 4
- 2021Trap of Feature Diversity in the Learning of MLPscited by 4
Built on by
Citing papers per year: 2022: 1; 2023: 0; 2024: 0; 2025: 1.
2 papers.