# Why do These Match? Explaining the Behavior of Image Similarity Models

arXiv: https://arxiv.org/abs/1905.10797 · HTML: https://mlsanity.com/paper/1905.10797/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 May 2019.

## Built on by (most cited first, 10 of 10)

- [Overlooked factors in concept-based explanations: Dataset choice, concept learnability, and human capability](https://mlsanity.com/paper/2207.09615/) (2022)
- [Effectively Leveraging Attributes for Visual Similarity](https://mlsanity.com/paper/2105.01695/) (2021)
- [Learning Visual Explanations for DCNN-Based Image Classifiers Using an Attention Mechanism](https://mlsanity.com/paper/2209.11189/) (2022)
- [Self-learn to Explain Siamese Networks Robustly](https://mlsanity.com/paper/2109.07371/) (2021)
- [A clinically motivated self-supervised approach for content-based image retrieval of CT liver images](https://mlsanity.com/paper/2207.04812/) (2022)
- [Explaining Caption-Image Interactions in CLIP Models with Second-Order Attributions](https://mlsanity.com/paper/2408.14153/) (2024)
- [Analogies and Feature Attributions for Model Agnostic Explanation of Similarity Learners](https://mlsanity.com/paper/2202.01153/) (2022)
- [S-Extension Patch: A simple and efficient way to extend an object detection model](https://mlsanity.com/paper/2110.02670/) (2021)
- [Dataset Structural Index: Leveraging a machine's perspective towards visual data](https://mlsanity.com/paper/2110.04070/) (2021)
- [Explaining Image Similarity with Automatically Extracted Concept Activation Vectors](https://mlsanity.com/paper/2607.28386/) (2026)

## Abstract (arXiv)

Explaining a deep learning model can help users understand its behavior and allow researchers to discern its shortcomings. Recent work has primarily focused on explaining models for tasks like image classification or visual question answering. In this paper, we introduce Salient Attributes for Network Explanation (SANE) to explain image similarity models, where a model's output is a score measuring the similarity of two inputs rather than a classification score. In this task, an explanation depends on both of the input images, so standard methods do not apply. Our SANE explanations pairs a saliency map identifying important image regions with an attribute that best explains the match. We find that our explanations provide additional information not typically captured by saliency maps alone, and can also improve performance on the classic task of attribute recognition. Our approach's ability to generalize is demonstrated on two datasets from diverse domains, Polyvore Outfits and Animals with Attributes 2. Code available at: https://github.com/VisionLearningGroup/SANE