mlsanityData to 7 Oct 2026

From Feature Visualization to Visual Circuits: Effect of Adversarial Model Manipulation

arXiv 2406.01365 ↗First posted 3 Jun 2024cs.CV cs.CR cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Understanding the inner working functionality of large-scale deep neural networks is challenging yet crucial in several high-stakes applications. Mechanistic inter- pretability is an emergent field that tackles this challenge, often by identifying human-understandable subgraphs in deep neural networks known as circuits. In vision-pretrained models, these subgraphs are usually interpreted by visualizing their node features through a popular technique called feature visualization. Recent works have analyzed the stability of different feature visualization types under the adversarial model manipulation framework. This paper starts by addressing limitations in existing works by proposing a novel attack called ProxPulse that simultaneously manipulates the two types of feature visualizations. Surprisingly, when analyzing these attacks under the umbrella of visual circuits, we find that visual circuits show some robustness to ProxPulse. We, therefore, introduce a new attack based on ProxPulse that unveils the manipulability of visual circuits, shedding light on their lack of robustness. The effectiveness of these attacks is validated using pre-trained AlexNet and ResNet-50 models on ImageNet.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 28 of its 38 references that are in the map. First posted on arXiv 3 Jun 2024.

Builds on

28 of 38 references.

References · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2016Densely Connected Convolutional Networkscited by 6,691
  5. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  6. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  7. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  8. 2018Sanity Checks for Saliency Mapscited by 1,149
  9. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  10. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  11. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  12. 2021Knowledge Neurons in Pretrained Transformerscited by 590
16 more
  1. 2014Understanding Deep Image Representations by Inverting Themcited by 399
  2. 2015Understanding Neural Networks Through Deep Visualizationcited by 332
  3. 2020Understanding the Role of Individual Units in a Deep Neural Networkcited by 254
  4. 2019Explanations can be manipulated and geometry is to blamecited by 215
  5. 2022CLIP-Dissect: Automatic Description of Neuron Representations in Deep Vision Networkscited by 144
  6. 2023A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operationscited by 141
  7. 2022Natural Language Descriptions of Deep Visual Featurescited by 136
  8. 2013Stabilitycited by 119
  9. 2019Fooling Neural Network Interpretations via Adversarial Model Manipulationcited by 112
  10. 2018Decoupling Direction and Norm for Efficient Gradient-Based L2 Adversarial Attacks and Defensescited by 109
  11. 2020Fairwashing Explanations with Off-Manifold Detergentcited by 71
  12. 2023Don't trust your eyes: on the (un)reliability of feature visualizationscited by 32
  13. 2021How Well do Feature Visualizations Support Causal Understanding of CNN Activations?cited by 28
  14. 2023Scale Alone Does not Improve Mechanistic Interpretability in Vision Modelscited by 17
  15. 2023Adversarial Attacks on the Interpretation of Neuron Activation Maximizationcited by 7
  16. 2024Manipulating Feature Visualizations with Gradient Slingshotscited by 3

Built on by

Not cited yet.