# Extracting Causal Visual Features for Limited label Classification

arXiv: https://arxiv.org/abs/2103.12322 · HTML: https://mlsanity.com/paper/2103.12322/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 25 references that are in the map.
- First posted on arXiv 23 Mar 2021.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks](https://mlsanity.com/paper/1710.11063/) (2017)
- [RISE: Randomized Input Sampling for Explanation of Black-box Models](https://mlsanity.com/paper/1806.07421/) (2018)
- [COVID-CT-Dataset: A CT Scan Dataset about COVID-19](https://mlsanity.com/paper/2003.13865/) (2020)
- [Discovering Causal Signals in Images](https://mlsanity.com/paper/1605.08179/) (2016)
- [Gradients as a Measure of Uncertainty in Neural Networks](https://mlsanity.com/paper/2008.08030/) (2020)
- [Backpropagated Gradient Representations for Anomaly Detection](https://mlsanity.com/paper/2007.09507/) (2020)
- [A Machine Learning Benchmark for Facies Classification](https://mlsanity.com/paper/1901.07659/) (2019)
- [Contrastive Explanations in Neural Networks](https://mlsanity.com/paper/2008.00178/) (2020)
- [Distorted Representation Space Characterization Through Backpropagated Gradients](https://mlsanity.com/paper/1908.09998/) (2019)
- [Implicit Saliency in Deep Neural Networks](https://mlsanity.com/paper/2008.01874/) (2020)
- [Relative Afferent Pupillary Defect Screening through Transfer Learning](https://mlsanity.com/paper/1908.02300/) (2019)

## Built on by (most cited first, 12 of 12)

- [OLIVES Dataset: Ophthalmic Labels for Investigating Visual Eye Semantics](https://mlsanity.com/paper/2209.11195/) (2022)
- [Introspective Learning : A Two-Stage Approach for Inference in Neural Networks](https://mlsanity.com/paper/2209.08425/) (2022)
- [Gradient-Based Adversarial and Out-of-Distribution Detection](https://mlsanity.com/paper/2206.08255/) (2022)
- [Explanatory Paradigms in Neural Networks](https://mlsanity.com/paper/2202.11838/) (2022)
- [Patient Aware Active Learning for Fine-Grained OCT Classification](https://mlsanity.com/paper/2206.11485/) (2022)
- [Probing the Purview of Neural Networks via Gradient Analysis](https://mlsanity.com/paper/2304.02834/) (2023)
- [VOICE: Variance of Induced Contrastive Explanations to quantify Uncertainty in Neural Network Interpretability](https://mlsanity.com/paper/2406.00573/) (2024)
- [Stochastic Surprisal: An inferential measurement of Free Energy in Neural Networks](https://mlsanity.com/paper/2302.05776/) (2023)
- [Counterfactual Gradients-based Quantification of Prediction Trust in Neural Networks](https://mlsanity.com/paper/2405.13758/) (2024)
- [Are Objective Explanatory Evaluation metrics Trustworthy? An Adversarial Analysis](https://mlsanity.com/paper/2406.07820/) (2024)
- [Learning Trajectory-Conditioned Relations to Predict Pedestrian Crossing Behavior](https://mlsanity.com/paper/2301.05796/) (2023)
- [Explaining Representation Learning with Perceptual Components](https://mlsanity.com/paper/2406.06930/) (2024)

## Abstract (arXiv)

Neural networks trained to classify images do so by identifying features that allow them to distinguish between classes. These sets of features are either causal or context dependent. Grad-CAM is a popular method of visualizing both sets of features. In this paper, we formalize this feature divide and provide a methodology to extract causal features from Grad-CAM. We do so by defining context features as those features that allow contrast between predicted class and any contrast class. We then apply a set theoretic approach to separate causal from contrast features for COVID-19 CT scans. We show that on average, the image regions with the proposed causal features require 15% less bits when encoded using Huffman encoding, compared to Grad-CAM, for an average increase of 3% classification accuracy, over Grad-CAM. Moreover, we validate the transfer-ability of causal features between networks and comment on the non-human interpretable causal nature of current networks.