mlsanityData to 7 Oct 2026

Layerwise Change of Knowledge in Neural Networks

arXiv 2409.08712 ↗First posted 13 Sep 2024cs.LG cs.AI cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

This paper aims to explain how a deep neural network (DNN) gradually extracts new knowledge and forgets noisy features through layers in forward propagation. Up to now, although the definition of knowledge encoded by the DNN has not reached a consensus, Previous studies have derived a series of mathematical evidence to take interactions as symbolic primitive inference patterns encoded by a DNN. We extend the definition of interactions and, for the first time, extract interactions encoded by intermediate layers. We quantify and track the newly emerged interactions and the forgotten interactions in each layer during the forward propagation, which shed new light on the learning behavior of DNNs. The layer-wise change of interactions also reveals the change of the generalization capacity and instability of feature representations of a DNN.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 31 of its 43 references that are in the map. First posted on arXiv 13 Sep 2024.

Builds on

31 of 43 references.

References · 31

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  5. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  6. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
  7. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  8. 2018This Looks Like That: Deep Learning for Interpretable Image Recognitioncited by 731
  9. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 703
  10. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  11. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  12. 2019Understanding Deep Networks via Extremal Perturbations and Smooth Maskscited by 257
19 more
  1. 2017Real Time Image Saliency for Black Box Classifierscited by 246
  2. 2020Information-Theoretic Probing for Linguistic Structurecited by 204
  3. 2018Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approachcited by 181
  4. 2019The Shapley Taylor Interaction Indexcited by 143
  5. 2019Explaining Deep Neural Networks with a Polynomial Time Algorithm for Shapley Values Approximationcited by 115
  6. 2020Relative Flatness and Generalizationcited by 88
  7. 2021Discovering and Explaining the Representation Bottleneck of DNNscited by 42
  8. 2021Defining and Quantifying the Emergence of Sparse Concepts in DNNscited by 36
  9. 2022Missingness Bias in Model Debuggingcited by 36
  10. 2018Caveats for information bottleneck in deterministic scenarioscited by 34
  11. 2021PAC-Bayes Information Bottleneckcited by 28
  12. 2023Does a Neural Network Really Encode Symbolic Concepts?cited by 27
  13. 2023Explaining Generalization Power of a DNN Using Interactive Conceptscited by 25
  14. 2019Interpretable CNNs for Object Classificationcited by 16
  15. 2019Knowledge Consistency between Neural Networks and Beyondcited by 16
  16. 2021Interpretable Compositional Convolutional Neural Networkscited by 15
  17. 2023Can the Inference Logic of Large Language Models be Disentangled into Symbolic Concepts?cited by 11
  18. 2023Technical Note: Defining and Quantifying AND-OR Interactions for Faithful and Concise Explanation of DNNscited by 10
  19. 2023Explaining How a Neural Network Play the Go Game and Let People Learncited by 6

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 1.

4 papers.

  1. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  2. 2024Why pre-training is beneficial for downstream classification tasks?cited by 1
  3. 2025Technical Report: Quantifying and Analyzing the Generalization Power of a DNNcited by 1
  4. 2024Disentangling Regional Primitives for Image Generationcited by 0