# Decompose-and-Compose: A Compositional Approach to Mitigating Spurious Correlation arXiv: https://arxiv.org/abs/2402.18919 ยท HTML: https://mlsanity.com/paper/2402.18919/ ## Key facts - Cited by 18 papers in the map (data through 14 Oct 2026). - Builds on 31 of its 48 references that are in the map. - First posted on arXiv 29 Feb 2024. ## Builds on (most cited first, 30 of 31) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016) - [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017) - [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014) - [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019) - [Out-of-Distribution Generalization via Risk Extrapolation (REx)](https://mlsanity.com/paper/2003.00688/) (2020) - [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018) - [Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks](https://mlsanity.com/paper/1910.01279/) (2019) - [Just Train Twice: Improving Group Robustness without Training Group Information](https://mlsanity.com/paper/2107.09044/) (2021) - [Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations](https://mlsanity.com/paper/2204.02937/) (2022) - [The Pitfalls of Simplicity Bias in Neural Networks](https://mlsanity.com/paper/2006.07710/) (2020) - [An Investigation of Why Overparameterization Exacerbates Spurious Correlations](https://mlsanity.com/paper/2005.04345/) (2020) - [Adversarial Domain Adaptation with Domain Mixup](https://mlsanity.com/paper/1912.01805/) (2019) - [Improving Out-of-Distribution Robustness via Selective Augmentation](https://mlsanity.com/paper/2201.00299/) (2022) - [Fishr: Invariant Gradient Variances for Out-of-Distribution Generalization](https://mlsanity.com/paper/2109.02934/) (2021) - [Simple data balancing achieves competitive worst-group-accuracy](https://mlsanity.com/paper/2110.14503/) (2021) - [Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlations](https://mlsanity.com/paper/2203.01517/) (2022) - [Tell Me Where to Look: Guided Attention Inference Network](https://mlsanity.com/paper/1802.10171/) (2018) - [Learning from Failure: Training Debiased Classifier from Biased Classifier](https://mlsanity.com/paper/2007.02561/) (2020) - [Fair Attribute Classification through Latent Space De-biasing](https://mlsanity.com/paper/2012.01469/) (2020) - [OoD-Bench: Quantifying and Understanding Two Dimensions of Out-of-Distribution Generalization](https://mlsanity.com/paper/2106.03721/) (2021) - [MetaShift: A Dataset of Datasets for Evaluating Contextual Distribution Shifts and Training Conflicts](https://mlsanity.com/paper/2202.06523/) (2022) - [Causal Attention for Unbiased Visual Recognition](https://mlsanity.com/paper/2108.08782/) (2021) - [Discover and Cure: Concept-aware Mitigation of Spurious Correlation](https://mlsanity.com/paper/2305.00650/) (2023) - [BiaSwap: Removing dataset bias with bias-tailored swapping augmentation](https://mlsanity.com/paper/2108.10008/) (2021) - [Agree to Disagree: Diversity through Disagreement for Better Transferability](https://mlsanity.com/paper/2202.04414/) (2022) - [Simple and Fast Group Robustness by Automatic Feature Reweighting](https://mlsanity.com/paper/2306.11074/) (2023) - [MaskTune: Mitigating Spurious Correlations by Forcing to Explore](https://mlsanity.com/paper/2210.00055/) (2022) - [Masked Images Are Counterfactual Samples for Robust Fine-tuning](https://mlsanity.com/paper/2303.03052/) (2023) - [A Closer Look at Model Adaptation using Feature Distortion and Simplicity Bias](https://mlsanity.com/paper/2303.13500/) (2023) ## Built on by (most cited first, 18 of 18) - [PatchGuard: Adversarially Robust Anomaly Detection and Localization through Vision Transformers and Pseudo Anomalies](https://mlsanity.com/paper/2506.09237/) (2025) - [DDB: Diffusion Driven Balancing to Address Spurious Correlations](https://mlsanity.com/paper/2503.17226/) (2025) - [LibraGrad: Balancing Gradient Flow for Universally Better Vision Transformer Attributions](https://mlsanity.com/paper/2411.16760/) (2024) - [GHOST: Hallucination-Inducing Image Generation for Multimodal LLMs](https://mlsanity.com/paper/2509.25178/) (2025) - [Interpretable 3D Neural Object Volumes for Robust Conceptual Reasoning](https://mlsanity.com/paper/2503.13429/) (2025) - [Do Students Debias Like Teachers? On the Distillability of Bias Mitigation Methods](https://mlsanity.com/paper/2510.26038/) (2025) - [Single-weight Model Editing for Post-hoc Spurious Correlation Neutralization](https://mlsanity.com/paper/2501.14182/) (2025) - [Fine-Grained Class-Conditional Distribution Balancing for Debiased Learning](https://mlsanity.com/paper/2505.06831/) (2025) - [Two-stage Vision Transformers and Hard Masking offer Robust Object Representations](https://mlsanity.com/paper/2506.08915/) (2025) - [Robust Context-Aware Object Recognition](https://mlsanity.com/paper/2510.00618/) (2025) - [HSFM: Hard-Set-Guided Feature-Space Meta-Learning for Robust Classification under Spurious Correlations](https://mlsanity.com/paper/2603.29313/) (2026) - [Trained Models Tell Us How to Make Them Robust to Spurious Correlation without Group Annotation](https://mlsanity.com/paper/2410.05345/) (2024) - [Spatially Grounded Concept-Based Image Classification](https://mlsanity.com/paper/2510.04180/) (2025) - [Mitigating Shortcut Learning via Feature Disentanglement in Medical Imaging: A Benchmark Study](https://mlsanity.com/paper/2602.18502/) (2026) - [CoDA: Exploring Chain-of-Distribution Attacks and Post-Hoc Token-Space Repair for Medical Vision-Language Models](https://mlsanity.com/paper/2603.18545/) (2026) - [ContextShift: A Controlled Benchmark for Context Dependence in Object Detection](https://mlsanity.com/paper/2606.09495/) (2026) - [Breaking Spurious Correlations via Generative Randomization and Cross-Variant Self-Supervised Learning](https://mlsanity.com/paper/2607.05850/) (2026) - [SAGE: Subpopulation-Aware Generative Enhancement for Mitigating Spurious Correlations](https://mlsanity.com/paper/2609.01051/) (2026) ## Abstract (arXiv) While standard Empirical Risk Minimization (ERM) training is proven effective for image classification on in-distribution data, it fails to perform well on out-of-distribution samples. One of the main sources of distribution shift for image classification is the compositional nature of images. Specifically, in addition to the main object or component(s) determining the label, some other image components usually exist, which may lead to the shift of input distribution between train and test environments. More importantly, these components may have spurious correlations with the label. To address this issue, we propose Decompose-and-Compose (DaC), which improves robustness to correlation shift by a compositional approach based on combining elements of images. Based on our observations, models trained with ERM usually highly attend to either the causal components or the components having a high spurious correlation with the label (especially in datapoints on which models have a high confidence). In fact, according to the amount of spurious correlation and the easiness of classification based on the causal or non-causal components, the model usually attends to one of these more (on samples with high confidence). Following this, we first try to identify the causal components of images using class activation maps of models trained with ERM. Afterward, we intervene on images by combining them and retraining the model on the augmented data, including the counterfactual ones. Along with its high interpretability, this work proposes a group-balancing method by intervening on images without requiring group labels or information regarding the spurious features during training. The method has an overall better worst group accuracy compared to previous methods with the same amount of supervision on the group labels in correlation shift.