# Mitigating the Effect of Incidental Correlations on Part-based Learning

arXiv: https://arxiv.org/abs/2310.00377 · HTML: https://mlsanity.com/paper/2310.00377/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 59 references that are in the map.
- First posted on arXiv 30 Sep 2023.

## Builds on (most cited first, 30 of 32)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [Meta-Learning for Semi-Supervised Few-Shot Classification](https://mlsanity.com/paper/1803.00676/) (2018)
- [TADAM: Task dependent adaptive metric for improved few-shot learning](https://mlsanity.com/paper/1805.10123/) (2018)
- [Meta-learning with differentiable closed-form solvers](https://mlsanity.com/paper/1805.08136/) (2018)
- [Noise or Signal: The Role of Image Backgrounds in Object Recognition](https://mlsanity.com/paper/2006.09994/) (2020)
- [Tagger: Deep Unsupervised Perceptual Grouping](https://mlsanity.com/paper/1606.06724/) (2016)
- [Unsupervised Part-Based Disentangling of Object Shape and Appearance](https://mlsanity.com/paper/1903.06946/) (2019)
- [Rectifying the Shortcut Learning of Background for Few-Shot Learning](https://mlsanity.com/paper/2107.07746/) (2021)
- [Learning Compositional Representations for Few-Shot Recognition](https://mlsanity.com/paper/1812.09213/) (2018)
- [Not Using the Car to See the Sidewalk: Quantifying and Controlling the Effects of Context in Classification and Segmentation](https://mlsanity.com/paper/1812.06707/) (2018)
- [Few-Shot Learning via Saliency-guided Hallucination of Samples](https://mlsanity.com/paper/1904.03472/) (2019)
- [Challenges of Big Data Analysis](https://mlsanity.com/paper/1308.1479/) (2013)
- [Rethinking Generalization in Few-Shot Classification](https://mlsanity.com/paper/2206.07267/) (2022)
- [How to Train Vision Transformer on Small-scale Datasets?](https://mlsanity.com/paper/2210.07240/) (2022)
- [Attribute Surrogates Learning and Spectral Tokens Pooling in Transformers for Few-shot Learning](https://mlsanity.com/paper/2203.09064/) (2022)
- [Matching Feature Sets for Few-Shot Image Classification](https://mlsanity.com/paper/2204.00949/) (2022)
- [Combining Compositional Models and Deep Networks For Robust Object Classification under Occlusion](https://mlsanity.com/paper/1905.11826/) (2019)
- [Mixture-based Feature Space Learning for Few-shot Image Classification](https://mlsanity.com/paper/2011.11872/) (2020)
- [Supervised Masked Knowledge Distillation for Few-Shot Transformers](https://mlsanity.com/paper/2303.15466/) (2023)
- [Self-Promoted Supervision for Few-Shot Transformer](https://mlsanity.com/paper/2203.07057/) (2022)
- [Unsupervised Discovery of Parts, Structure, and Dynamics](https://mlsanity.com/paper/1903.05136/) (2019)
- [Channel Importance Matters in Few-Shot Image Classification](https://mlsanity.com/paper/2206.08126/) (2022)
- [Few-Shot Learning Meets Transformer: Unified Query-Support Transformers for Few-Shot Classification](https://mlsanity.com/paper/2208.12398/) (2022)
- [Rectify ViT Shortcut Learning by Visual Saliency](https://mlsanity.com/paper/2206.08567/) (2022)
- [Unsupervised Co-part Segmentation through Assembly](https://mlsanity.com/paper/2106.05897/) (2021)
- [Label Hallucination for Few-Shot Classification](https://mlsanity.com/paper/2112.03340/) (2021)

## Built on by (most cited first, 6 of 6)

- [TIBET: Identifying and Evaluating Biases in Text-to-Image Generative Models](https://mlsanity.com/paper/2312.01261/) (2023)
- [Animal Identification with Independent Foreground and Background Modeling](https://mlsanity.com/paper/2408.12930/) (2024)
- [FungiTastic: A multi-modal dataset and benchmark for image categorization](https://mlsanity.com/paper/2408.13632/) (2024)
- [Efficient Calisthenics Skills Classification through Foreground Instance Selection and Depth Estimation](https://mlsanity.com/paper/2507.12292/) (2025)
- [Robust Context-Aware Object Recognition](https://mlsanity.com/paper/2510.00618/) (2025)
- [Bringing the Context Back into Object Recognition, Robustly](https://mlsanity.com/paper/2411.15933/) (2024)

## Abstract (arXiv)

Intelligent systems possess a crucial characteristic of breaking complicated problems into smaller reusable components or parts and adjusting to new tasks using these part representations. However, current part-learners encounter difficulties in dealing with incidental correlations resulting from the limited observations of objects that may appear only in specific arrangements or with specific backgrounds. These incidental correlations may have a detrimental impact on the generalization and interpretability of learned part representations. This study asserts that part-based representations could be more interpretable and generalize better with limited data, employing two innovative regularization methods. The first regularization separates foreground and background information's generative process via a unique mixture-of-parts formulation. Structural constraints are imposed on the parts using a weakly-supervised loss, guaranteeing that the mixture-of-parts for foreground and background entails soft, object-agnostic masks. The second regularization assumes the form of a distillation loss, ensuring the invariance of the learned parts to the incidental background correlations. Furthermore, we incorporate sparse and orthogonal constraints to facilitate learning high-quality part representations. By reducing the impact of incidental background correlations on the learned parts, we exhibit state-of-the-art (SoTA) performance on few-shot learning tasks on benchmark datasets, including MiniImagenet, TieredImageNet, and FC100. We also demonstrate that the part-based representations acquired through our approach generalize better than existing techniques, even under domain shifts of the background and common data corruption on the ImageNet-9 dataset. The implementation is available on GitHub: https://github.com/GauravBh1010tt/DPViT.git