mlsanityData to 14 Oct 2026

Understanding Activation Patterns in Artificial Neural Networks by Exploring Stochastic Processes

arXiv 2308.00858 ↗First posted 1 Aug 2023cs.LG cs.AI cs.CCcited by 0

Not on the map. Why?

Abstract (arXiv)

To gain a deeper understanding of the behavior and learning dynamics of (deep) artificial neural networks, it is valuable to employ mathematical abstractions and models. These tools provide a simplified perspective on network performance and facilitate systematic investigations through simulations. In this paper, we propose utilizing the framework of stochastic processes, which has been underutilized thus far. Our approach models activation patterns of thresholded nodes in (deep) artificial neural networks as stochastic processes. We focus solely on activation frequency, leveraging neuroscience techniques used for real neuron spike trains. During a classification task, we extract spiking activity and use an arrival process following the Poisson distribution. We examine observed data from various artificial neural networks in image recognition tasks, fitting the proposed model's assumptions. Through this, we derive parameters describing activation patterns in each network. Our analysis covers randomly initialized, generalizing, and memorizing networks, revealing consistent differences across architectures and training sets. Calculating Mean Firing Rate, Mean Fano Factor, and Variances, we find stable indicators of memorization during learning, providing valuable insights into network behavior. The proposed model shows promise in describing activation patterns and could serve as a general framework for future investigations. It has potential applications in theoretical simulations, pruning, and transfer learning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 37 of its 87 references that are in the map. First posted on arXiv 1 Aug 2023.

Builds on

37 of 87 references.

References · 37

  1. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  2. 2016Densely Connected Convolutional Networkscited by 6,720
  3. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
  4. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  5. 2018Sanity Checks for Saliency Mapscited by 1,156
  6. 2017A Closer Look at Memorization in Deep Networkscited by 942
  7. 2014Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Imagescited by 776
  8. 2017Exploring Generalization in Deep Learningcited by 600
  9. 2020What is the State of Neural Network Pruning?cited by 583
  10. 2015Understanding Neural Networks Through Deep Visualizationcited by 334
  11. 2016Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architecturescited by 221
  12. 2016Synthesizing the preferred inputs for neurons in neural networks via deep generator networkscited by 181
25 more
  1. 2021The Principles of Deep Learning Theorycited by 173
  2. 2019Deep ReLU Networks Have Surprisingly Few Activation Patternscited by 161
  3. 2015Structured Pruning of Deep Convolutional Neural Networkscited by 149
  4. 2016Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networkscited by 106
  5. 2021On the geometry of generalization and memorization in deep neural networkscited by 73
  6. 2021The Modern Mathematics of Deep Learningcited by 63
  7. 2019Regularizing Activation Distribution for Training Binarized Deep Networkscited by 61
  8. 2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 44
  9. 2020How does Weight Correlation Affect the Generalisation Ability of Deep Neural Networkscited by 32
  10. 2022DropNet: Reducing Neural Network Complexity via Iterative Pruningcited by 29
  11. 2023Neuron Activation Coverage: Rethinking Out-of-distribution Detection and Generalizationcited by 28
  12. 2019Quantifying the generalization error in deep learning in terms of data distribution and neural network smoothnesscited by 22
  13. 2017Choosing Between Methods of Combining p-valuescited by 17
  14. 2018DNN or k-NN: That is the Generalize vs. Memorize Questioncited by 16
  15. 2020NeuralSens: Sensitivity Analysis of Neural Networkscited by 14
  16. 2021On Contrastive Representations of Stochastic Processescited by 12
  17. 2020Point process models for sequence detection in high-dimensional neural spike trainscited by 10
  18. 2017Revisiting Activation Regularization for Language RNNscited by 6
  19. 2019Abstraction Mechanisms Predict Generalization in Deep Neural Networkscited by 5
  20. 2020Regularizing activations in neural networks via distribution matching with the Wasserstein metriccited by 4
  21. 2019Towards Characterizing and Limiting Information Exposure in DNN Layerscited by 3
  22. 2022Adaptive Activation-based Structured Pruningcited by 3
  23. 2019Towards Expressive Priors for Bayesian Neural Networks: Poisson Process Radial Basis Function Networkscited by 2
  24. 2022Improving Meta-Learning Generalization with Activation-Based Early-Stoppingcited by 2
  25. 2020Signal Fluctuations and the Information Transmission Rates in Binary Communication Channelscited by 1

Built on by

Not cited yet.