# Adversarial TCAV -- Robust and Effective Interpretation of Intermediate Layers in Neural Networks

arXiv: https://arxiv.org/abs/2002.03549 · HTML: https://mlsanity.com/paper/2002.03549/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Feb 2020.

## Built on by (most cited first, 12 of 12)

- [Getting aligned on representational alignment](https://mlsanity.com/paper/2310.13018/) (2023)
- [Concept Activation Regions: A Generalized Framework For Concept-Based Explanations](https://mlsanity.com/paper/2209.11222/) (2022)
- [Bridging the Human-AI Knowledge Gap: Concept Discovery and Transfer in AlphaZero](https://mlsanity.com/paper/2310.16410/) (2023)
- [Understanding and Enhancing Robustness of Concept-based Models](https://mlsanity.com/paper/2211.16080/) (2022)
- [Concept Distillation: Leveraging Human-Centered Explanations for Model Improvement](https://mlsanity.com/paper/2311.15303/) (2023)
- [LG-CAV: Train Any Concept Activation Vector with Language Guidance](https://mlsanity.com/paper/2410.10308/) (2024)
- [A survey on Concept-based Approaches For Model Improvement](https://mlsanity.com/paper/2403.14566/) (2024)
- [On The Variability of Concept Activation Vectors](https://mlsanity.com/paper/2509.24058/) (2025)
- [Towards interpreting ML-based automated malware detection models: a survey](https://mlsanity.com/paper/2101.06232/) (2021)
- [Concept Boundary Vectors](https://mlsanity.com/paper/2412.15698/) (2024)
- [Probing the Probes: Methods and Metrics for Concept Alignment](https://mlsanity.com/paper/2511.04312/) (2025)
- [A High-Resolution Landscape Dataset for Concept-Based XAI With Application to Species Distribution Models](https://mlsanity.com/paper/2604.13240/) (2026)

## Abstract (arXiv)

Interpreting neural network decisions and the information learned in intermediate layers is still a challenge due to the opaque internal state and shared non-linear interactions. Although (Kim et al, 2017) proposed to interpret intermediate layers by quantifying its ability to distinguish a user-defined concept (from random examples), the questions of robustness (variation against the choice of random examples) and effectiveness (retrieval rate of concept images) remain. We investigate these two properties and propose improvements to make concept activations reliable for practical use. Effectiveness: If the intermediate layer has effectively learned a user-defined concept, it should be able to recall --- at the testing step --- most of the images containing the proposed concept. For instance, we observed that the recall rate of Tiger shark and Great white shark from the ImageNet dataset with "Fins" as a user-defined concept was only 18.35% for VGG16. To increase the effectiveness of concept learning, we propose A-CAV --- the Adversarial Concept Activation Vector --- this results in larger margins between user concepts and (negative) random examples. This approach improves the aforesaid recall to 76.83% for VGG16. For robustness, we define it as the ability of an intermediate layer to be consistent in its recall rate (the effectiveness) for different random seeds. We observed that TCAV has a large variance in recalling a concept across different random seeds. For example, the recall of cat images (from a layer learning the concept of tail) varies from 18% to 86% with 20.85% standard deviation on VGG16. We propose a simple and scalable modification that employs a Gram-Schmidt process to sample random noise from concepts and learn an average "concept classifier". This approach improves the aforesaid standard deviation from 20.85% to 6.4%.