# PDiscoFormer: Relaxing Part Discovery Constraints with Vision Transformers

arXiv: https://arxiv.org/abs/2407.04538 · HTML: https://mlsanity.com/paper/2407.04538/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 48 references that are in the map.
- First posted on arXiv 5 Jul 2024.

## Builds on (most cited first, 30 of 30)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [This Looks Like That: Deep Learning for Interpretable Image Recognition](https://mlsanity.com/paper/1806.10574/) (2018)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [GroupViT: Semantic Segmentation Emerges from Text Supervision](https://mlsanity.com/paper/2202.11094/) (2022)
- [Noise or Signal: The Role of Image Backgrounds in Object Recognition](https://mlsanity.com/paper/2006.09994/) (2020)
- [Deep ViT Features as Dense Visual Descriptors](https://mlsanity.com/paper/2112.05814/) (2021)
- [One weird trick for parallelizing convolutional neural networks](https://mlsanity.com/paper/1404.5997/) (2014)
- [Localizing Objects with Self-Supervised Transformers and no Labels](https://mlsanity.com/paper/2109.14279/) (2021)
- [Neural Prototype Trees for Interpretable Fine-grained Image Recognition](https://mlsanity.com/paper/2012.02046/) (2020)
- [PartImageNet: A Large, High-Quality Dataset of Parts](https://mlsanity.com/paper/2112.00933/) (2021)
- [SCOPS: Self-Supervised Co-Part Segmentation](https://mlsanity.com/paper/1905.01298/) (2019)
- [ProtoPShare: Prototype Sharing for Interpretable Image Classification and Similarity Discovery](https://mlsanity.com/paper/2011.14340/) (2020)
- [Perceptual Grouping in Contrastive Vision-Language Models](https://mlsanity.com/paper/2210.09996/) (2022)
- [Object discovery and representation networks](https://mlsanity.com/paper/2203.08777/) (2022)
- [Interpretable and Accurate Fine-grained Recognition via Region Grouping](https://mlsanity.com/paper/2005.10411/) (2020)
- [Unsupervised Part Discovery from Contrastive Reconstruction](https://mlsanity.com/paper/2111.06349/) (2021)
- [This Looks Like That, Because ... Explaining Prototypes for Interpretable Image Recognition](https://mlsanity.com/paper/2011.02863/) (2020)
- [This looks more like that: Enhancing Self-Explaining Models by Prototypical Relevance Propagation](https://mlsanity.com/paper/2108.12204/) (2021)
- [PDiscoNet: Semantically consistent part discovery for fine-grained recognition](https://mlsanity.com/paper/2309.03173/) (2023)
- [Masking Strategies for Background Bias Removal in Computer Vision Models](https://mlsanity.com/paper/2308.12127/) (2023)
- [Total Variation Optimization Layers for Computer Vision](https://mlsanity.com/paper/2204.03643/) (2022)
- [PARTICUL: Part Identification with Confidence measure using Unsupervised Learning](https://mlsanity.com/paper/2206.13304/) (2022)

## Built on by (most cited first, 9 of 9)

- [Adaptive Part Learning for Fine-Grained Generalized Category Discovery: A Plug-and-Play Enhancement](https://mlsanity.com/paper/2507.06928/) (2025)
- [SUB: Benchmarking CBM Generalization via Synthetic Attribute Substitutions](https://mlsanity.com/paper/2507.23784/) (2025)
- [Exploring Interpretability for Visual Prompt Tuning with Cross-layer Concepts](https://mlsanity.com/paper/2503.06084/) (2025)
- [Two-stage Vision Transformers and Hard Masking offer Robust Object Representations](https://mlsanity.com/paper/2506.08915/) (2025)
- [Unsupervised Part Discovery via Descriptor-Based Masked Image Restoration with Optimized Constraints](https://mlsanity.com/paper/2507.11985/) (2025)
- [Keypoint Counting Classifiers: Turning Vision Transformers into Self-Explainable Models Without Training](https://mlsanity.com/paper/2512.17891/) (2025)
- [Metonymy in vision models undermines attention-based interpretability](https://mlsanity.com/paper/2605.06095/) (2026)
- [Spatially Grounded Concept Bottleneck Models via Part-Factorized Attention](https://mlsanity.com/paper/2606.04364/) (2026)
- [RATS! Patches Talk Through Registers: Emergent Parts in Register Attention Transformers](https://mlsanity.com/paper/2606.14701/) (2026)

## Abstract (arXiv)

Computer vision methods that explicitly detect object parts and reason on them are a step towards inherently interpretable models. Existing approaches that perform part discovery driven by a fine-grained classification task make very restrictive assumptions on the geometric properties of the discovered parts; they should be small and compact. Although this prior is useful in some cases, in this paper we show that pre-trained transformer-based vision models, such as self-supervised DINOv2 ViT, enable the relaxation of these constraints. In particular, we find that a total variation (TV) prior, which allows for multiple connected components of any size, substantially outperforms previous work. We test our approach on three fine-grained classification benchmarks: CUB, PartImageNet and Oxford Flowers, and compare our results to previously published methods as well as a re-implementation of the state-of-the-art method PDiscoNet with a transformer-based backbone. We consistently obtain substantial improvements across the board, both on part discovery metrics and the downstream classification task, showing that the strong inductive biases in self-supervised ViT models require to rethink the geometric priors that can be used for unsupervised part discovery.