mlsanityData to 14 Oct 2026

Visualizing the Emergence of Intermediate Visual Patterns in DNNs

arXiv 2111.03505 ↗First posted 5 Nov 2021cs.CV cs.AI cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

This paper proposes a method to visualize the discrimination power of intermediate-layer visual patterns encoded by a DNN. Specifically, we visualize (1) how the DNN gradually learns regional visual patterns in each intermediate layer during the training process, and (2) the effects of the DNN using non-discriminative patterns in low layers to construct disciminative patterns in middle/high layers through the forward propagation. Based on our visualization method, we can quantify knowledge points (i.e., the number of discriminative visual patterns) learned by the DNN to evaluate the representation capacity of the DNN. Furthermore, this method also provides new insights into signal-processing behaviors of existing deep-learning techniques, such as adversarial attacks and knowledge distillation.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 50 of its 85 references that are in the map. First posted on arXiv 5 Nov 2021.

Builds on

50 of 85 references.

References · 50

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  7. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  8. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  9. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  10. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  11. 2015Learning Deep Features for Discriminative Localizationcited by 2,436
  12. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
38 more
  1. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  3. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,319
  4. 2017Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networkscited by 939
  5. 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 629
  6. 2017Interpretable Explanations of Black Boxes by Meaningful Perturbationcited by 612
  7. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 503
  8. 2014Understanding Deep Image Representations by Inverting Themcited by 403
  9. 2015Understanding Neural Networks Through Deep Visualizationcited by 334
  10. 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 303
  11. 2017NormFace: L2 Hypersphere Embedding for Face Verificationcited by 237
  12. 2018Sensitivity and Generalization in Neural Networks: an Empirical Studycited by 189
  13. 2018Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approachcited by 182
  14. 2014Object Detectors Emerge in Deep Scene CNNscited by 179
  15. 2015Inverting Visual Representations with Convolutional Networkscited by 170
  16. 2017Visualizing Deep Neural Network Decisions: Prediction Difference Analysiscited by 166
  17. 2018Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networkscited by 142
  18. 2016Information Dropout: Learning Optimal Representations Through Noisy Computationcited by 132
  19. 2020The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Modelscited by 125
  20. 2019Explaining Deep Neural Networks with a Polynomial Time Algorithm for Shapley Values Approximationcited by 115
  21. 2018L-Shapley and C-Shapley: Efficient Model Interpretation for Structured Datacited by 100
  22. 2019Are All Layers Created Equal?cited by 94
  23. 2017Learning how to explain neural networks: PatternNet and PatternAttributioncited by 84
  24. 2020A Unified Approach to Interpreting and Boosting Adversarial Transferabilitycited by 71
  25. 2020Explaining Knowledge Distillation by Quantifying the Knowledgecited by 66
  26. 2020Efficient nonparametric statistical inference on population feature importance using Shapley valuescited by 61
  27. 2019Generalization bounds for deep convolutional neural networkscited by 59
  28. 2020CNN Explainer: Learning Convolutional Neural Networks with Interactive Visualizationcited by 57
  29. 2017von Mises-Fisher Mixture Model-based Deep learning: Application to Face Verificationcited by 46
  30. 2019Stiffness: A New Perspective on Generalization in Neural Networkscited by 46
  31. 2019The intriguing role of module criticality in the generalization of deep networkscited by 40
  32. 2020Interpreting and Boosting Dropout from a Game-Theoretic Viewcited by 34
  33. 2018On Tighter Generalization Bound for Deep Neural Networks: CNNs, ResNets, and Beyondcited by 24
  34. 2018How Many Samples are Needed to Estimate a Convolutional or Recurrent Neural Network?cited by 19
  35. 2019Knowledge Consistency between Neural Networks and Beyondcited by 16
  36. 2021A Game-Theoretic Taxonomy of Visual Concepts in DNNscited by 16
  37. 2021Interpreting Attributions and Interactions of Adversarial Attackscited by 13
  38. 2021A Hypothesis for the Aesthetic Appreciation in Neural Networkscited by 9

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 2; 2023: 1; 2024: 2; 2025: 1.

7 papers.

  1. 2024LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Modelcited by 138
  2. 2022Efficient Attribute Unlearning: Towards Selective Removal of Input Attributes from Feature Representationscited by 17
  3. 2023Towards Free Data Selection with General-Purpose Modelscited by 10
  4. 2021Towards General and Efficient Active Learningcited by 6
  5. 2025A Survey on Interpretability in Visual Recognitioncited by 2
  6. 2022SBPF: Sensitiveness Based Pruning Framework For Convolutional Neural Network On Image Classificationcited by 0
  7. 2024InFiConD: Interactive No-code Fine-tuning with Concept-based Knowledge Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021What Do Deep Nets Learn? Class-wise Patterns Revealed in the Input Space
  2. 2022Quantifying the Knowledge in a DNN to Explain Knowledge Distillation for Classification
  3. 2023Don't trust your eyes: on the (un)reliability of feature visualizations
  4. 2025SDAKD: Student Discriminator Assisted Knowledge Distillation for Super-Resolution Generative Adversarial Networks
  5. 2021Visualizing the embedding space to explain the effect of knowledge distillation
  6. 2025Foundations and Models in Modern Computer Vision: Key Building Blocks in Landmark Architectures
  7. 2026Relational Knowledge Distillation Brings DNN Representations Close Enough to Humans to Be Aligned Without Supervision
  8. 2022Graph-based Neural Modules to Inspect Attention-based Architectures: A Position Paper