# Towards Better Analysis of Deep Convolutional Neural Networks

arXiv: https://arxiv.org/abs/1604.07043 · HTML: https://mlsanity.com/paper/1604.07043/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Apr 2016.

## Built on by (most cited first, 30 of 51)

- [From Attribution Maps to Human-Understandable Explanations through Concept Relevance Propagation](https://mlsanity.com/paper/2206.03208/) (2022)
- [Explainable Artificial Intelligence: A Survey of Needs, Techniques, Applications, and Future Direction](https://mlsanity.com/paper/2409.00265/) (2024)
- [Foundation Models Meet Visualizations: Challenges and Opportunities](https://mlsanity.com/paper/2310.05771/) (2023)
- [How Does Attention Work in Vision Transformers? A Visual Analytics Attempt](https://mlsanity.com/paper/2303.13731/) (2023)
- [The State of the Art in Enhancing Trust in Machine Learning Models with the Use of Visualizations](https://mlsanity.com/paper/2212.11737/) (2022)
- [Aligning Eyes between Humans and Deep Neural Network through Interactive Attention Alignment](https://mlsanity.com/paper/2202.02838/) (2022)
- [Visual Analytics For Machine Learning: A Data Perspective Survey](https://mlsanity.com/paper/2307.07712/) (2023)
- [VisQA: X-raying Vision and Language Reasoning in Transformers](https://mlsanity.com/paper/2104.00926/) (2021)
- [Diagnosing Ensemble Few-Shot Classifiers](https://mlsanity.com/paper/2206.04372/) (2022)
- [VAC-CNN: A Visual Analytics System for Comparative Studies of Deep Convolutional Neural Networks](https://mlsanity.com/paper/2110.13252/) (2021)
- [Towards Visual Explainable Active Learning for Zero-Shot Classification](https://mlsanity.com/paper/2108.06730/) (2021)
- [Learning-From-Disagreement: A Model Comparison and Visual Analytics Framework](https://mlsanity.com/paper/2201.07849/) (2022)
- [Human-Guided Image Generation for Expanding Small-Scale Training Image Datasets](https://mlsanity.com/paper/2412.16839/) (2024)
- [G2uardFL: Safeguarding Federated Learning Against Backdoor Attacks through Attributed Client Graph Clustering](https://mlsanity.com/paper/2306.04984/) (2023)
- [Towards an Understanding and Explanation for Mixed-Initiative Artificial Scientific Text Detection](https://mlsanity.com/paper/2304.05011/) (2023)
- [A Unified Interactive Model Evaluation for Classification, Object Detection, and Instance Segmentation in Computer Vision](https://mlsanity.com/paper/2308.05168/) (2023)
- [Explainable Adversarial Attacks in Deep Neural Networks Using Activation Profiles](https://mlsanity.com/paper/2103.10229/) (2021)
- [A Visual Analytics System for Improving Attention-based Traffic Forecasting Models](https://mlsanity.com/paper/2208.04350/) (2022)
- [VERB: Visualizing and Interpreting Bias Mitigation Techniques for Word Representations](https://mlsanity.com/paper/2104.02797/) (2021)
- [SIM2REALVIZ: Visualizing the Sim2Real Gap in Robot Ego-Pose Estimation](https://mlsanity.com/paper/2109.11801/) (2021)
- [Visual Analysis of Neural Architecture Spaces for Summarizing Design Principles](https://mlsanity.com/paper/2208.09665/) (2022)
- [Mitigating Backdoors in Federated Learning with FLD](https://mlsanity.com/paper/2303.00302/) (2023)
- [Are We Closing the Loop Yet? Gaps in the Generalizability of VIS4ML Research](https://mlsanity.com/paper/2308.06290/) (2023)
- [EL-VIT: Probing Vision Transformer with Interactive Visualization](https://mlsanity.com/paper/2401.12666/) (2024)
- [DNN Modularization via Activation-Driven Training](https://mlsanity.com/paper/2411.01074/) (2024)
- [Illuminating LLM Coding Agents: Visual Analytics for Deeper Understanding and Enhancement](https://mlsanity.com/paper/2508.12555/) (2025)
- [DeepEverest: Accelerating Declarative Top-K Queries for Deep Neural Network Interpretation](https://mlsanity.com/paper/2104.02234/) (2021)
- [Testing Feedforward Neural Networks Training Programs](https://mlsanity.com/paper/2204.00694/) (2022)
- [SingVisio: Visual Analytics of Diffusion Model for Singing Voice Conversion](https://mlsanity.com/paper/2402.12660/) (2024)
- [FaceX: Understanding Face Attribute Classifiers through Summary Model Explanations](https://mlsanity.com/paper/2412.07313/) (2024)

## Abstract (arXiv)

Deep convolutional neural networks (CNNs) have achieved breakthrough performance in many pattern recognition tasks such as image classification. However, the development of high-quality deep models typically relies on a substantial amount of trial-and-error, as there is still no clear understanding of when and why a deep model works. In this paper, we present a visual analytics approach for better understanding, diagnosing, and refining deep CNNs. We formulate a deep CNN as a directed acyclic graph. Based on this formulation, a hybrid visualization is developed to disclose the multiple facets of each neuron and the interactions between them. In particular, we introduce a hierarchical rectangle packing algorithm and a matrix reordering algorithm to show the derived features of a neuron cluster. We also propose a biclustering-based edge bundling method to reduce visual clutter caused by a large number of connections between neurons. We evaluated our method on a set of CNNs and the results are generally favorable.