# Explaining Representation Learning with Perceptual Components

arXiv: https://arxiv.org/abs/2406.06930 · HTML: https://mlsanity.com/paper/2406.06930/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 28 references that are in the map.
- First posted on arXiv 11 Jun 2024.

## Builds on (most cited first, 17 of 17)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [RISE: Randomized Input Sampling for Explanation of Black-box Models](https://mlsanity.com/paper/1806.07421/) (2018)
- [Interpretable Explanations of Black Boxes by Meaningful Perturbation](https://mlsanity.com/paper/1704.03296/) (2017)
- [Understanding Deep Networks via Extremal Perturbations and Smooth Masks](https://mlsanity.com/paper/1910.08485/) (2019)
- [Revisiting Self-Supervised Visual Representation Learning](https://mlsanity.com/paper/1901.09005/) (2019)
- [A Large-Scale Car Dataset for Fine-Grained Categorization and Verification](https://mlsanity.com/paper/1506.08959/) (2015)
- [CURE-OR: Challenging Unreal and Real Environments for Object Recognition](https://mlsanity.com/paper/1810.08293/) (2018)
- [Label-Free Explainability for Unsupervised Models](https://mlsanity.com/paper/2203.01928/) (2022)
- [Extracting Causal Visual Features for Limited label Classification](https://mlsanity.com/paper/2103.12322/) (2021)
- [RELAX: Representation Learning Explainability](https://mlsanity.com/paper/2112.10161/) (2021)
- [Explanatory Paradigms in Neural Networks](https://mlsanity.com/paper/2202.11838/) (2022)
- [FOCAL: A Cost-Aware Video Dataset for Active Learning](https://mlsanity.com/paper/2311.10591/) (2023)
- [Assessing The Importance Of Colours For CNNs In Object Recognition](https://mlsanity.com/paper/2012.06917/) (2020)
- [Contrastive Corpus Attribution for Explaining Representations](https://mlsanity.com/paper/2210.00107/) (2022)

## Abstract (arXiv)

Self-supervised models create representation spaces that lack clear semantic meaning. This interpretability problem of representations makes traditional explainability methods ineffective in this context. In this paper, we introduce a novel method to analyze representation spaces using three key perceptual components: color, shape, and texture. We employ selective masking of these components to observe changes in representations, resulting in distinct importance maps for each. In scenarios, where labels are absent, these importance maps provide more intuitive explanations as they are integral to the human visual system. Our approach enhances the interpretability of the representation space, offering explanations that resonate with human visual perception. We analyze how different training objectives create distinct representation spaces using perceptual components. Additionally, we examine the representation of images across diverse image domains, providing insights into the role of these components in different contexts.