# Consistent Explanations by Contrastive Learning

arXiv: https://arxiv.org/abs/2110.00527 · HTML: https://mlsanity.com/paper/2110.00527/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 47 references that are in the map.
- First posted on arXiv 1 Oct 2021.

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [RISE: Randomized Input Sampling for Explanation of Black-box Models](https://mlsanity.com/paper/1806.07421/) (2018)
- [This Looks Like That: Deep Learning for Interpretable Image Recognition](https://mlsanity.com/paper/1806.10574/) (2018)
- [Interpretable Explanations of Black Boxes by Meaningful Perturbation](https://mlsanity.com/paper/1704.03296/) (2017)
- [A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI](https://mlsanity.com/paper/1907.07374/) (2019)
- [The (Un)reliability of saliency methods](https://mlsanity.com/paper/1711.00867/) (2017)
- [Top-down Neural Attention by Excitation Backprop](https://mlsanity.com/paper/1608.00507/) (2016)
- [Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations](https://mlsanity.com/paper/1703.03717/) (2017)
- [Understanding Deep Networks via Extremal Perturbations and Smooth Masks](https://mlsanity.com/paper/1910.08485/) (2019)
- [Real Time Image Saliency for Black Box Classifiers](https://mlsanity.com/paper/1705.07857/) (2017)
- [Full-Gradient Representation for Neural Network Visualization](https://mlsanity.com/paper/1905.00780/) (2019)
- [Colorization as a Proxy Task for Visual Understanding](https://mlsanity.com/paper/1703.04044/) (2017)
- [Interpretations are useful: penalizing explanations to align neural networks with prior knowledge](https://mlsanity.com/paper/1909.13584/) (2019)

## Built on by (most cited first, 18 of 18)

- [Encoding Time-Series Explanations through Self-Supervised Model Behavior Consistency](https://mlsanity.com/paper/2306.02109/) (2023)
- [Improving Visual Grounding by Encouraging Consistent Gradient-based Explanations](https://mlsanity.com/paper/2206.15462/) (2022)
- [Are Data-driven Explanations Robust against Out-of-distribution Data?](https://mlsanity.com/paper/2303.16390/) (2023)
- [DEAL: Disentangle and Localize Concept-level Explanations for VLMs](https://mlsanity.com/paper/2407.14412/) (2024)
- [B-cosification: Transforming Deep Neural Networks to be Inherently Interpretable](https://mlsanity.com/paper/2411.00715/) (2024)
- [Visualizing and Understanding Contrastive Learning](https://mlsanity.com/paper/2206.09753/) (2022)
- [LICO: Explainable Models with Language-Image Consistency](https://mlsanity.com/paper/2310.09821/) (2023)
- [Allowing humans to interactively guide machines where to look does not always improve human-AI team's classification accuracy](https://mlsanity.com/paper/2404.05238/) (2024)
- [Mitigating Bias: Enhancing Image Classification by Improving Model Explanations](https://mlsanity.com/paper/2307.01473/) (2023)
- [ConPro: Learning Severity Representation for Medical Images using Contrastive Learning and Preference Optimization](https://mlsanity.com/paper/2404.18831/) (2024)
- [Attention IoU: Examining Biases in CelebA using Attention Maps](https://mlsanity.com/paper/2503.19846/) (2025)
- [Exploring the Lottery Ticket Hypothesis with Explainability Methods: Insights into Sparse Network Performance](https://mlsanity.com/paper/2307.13698/) (2023)
- [On the Consistency and Robustness of Saliency Explanations for Time Series Classification](https://mlsanity.com/paper/2309.01457/) (2023)
- [Where Not to Learn: Prior-Aligned Training with Subset-based Attribution Constraints](https://mlsanity.com/paper/2602.07008/) (2026)
- [Attri-Net: A Globally and Locally Inherently Interpretable Model for Multi-Label Classification Using Class-Specific Counterfactuals](https://mlsanity.com/paper/2406.05477/) (2024)
- [Hierarchical, Interpretable, Label-Free Concept Bottleneck Model](https://mlsanity.com/paper/2604.02468/) (2026)
- [Empirical Characterization of Rationale Stability Under Controlled Perturbations for Explainable Pattern Recognition](https://mlsanity.com/paper/2604.04456/) (2026)
- [Consistent Evidence, Robust Recognition: Faithful Attribution Regularization under Geometric Transformations](https://mlsanity.com/paper/2607.23835/) (2026)

## Abstract (arXiv)

Post-hoc explanation methods, e.g., Grad-CAM, enable humans to inspect the spatial regions responsible for a particular network decision. However, it is shown that such explanations are not always consistent with human priors, such as consistency across image transformations. Given an interpretation algorithm, e.g., Grad-CAM, we introduce a novel training method to train the model to produce more consistent explanations. Since obtaining the ground truth for a desired model interpretation is not a well-defined task, we adopt ideas from contrastive self-supervised learning, and apply them to the interpretations of the model rather than its embeddings. We show that our method, Contrastive Grad-CAM Consistency (CGC), results in Grad-CAM interpretation heatmaps that are more consistent with human annotations while still achieving comparable classification accuracy. Moreover, our method acts as a regularizer and improves the accuracy on limited-data, fine-grained classification settings. In addition, because our method does not rely on annotations, it allows for the incorporation of unlabeled data into training, which enables better generalization of the model. Our code is available here: https://github.com/UCDvision/CGC