# Leveraging Diffusion Disentangled Representations to Mitigate Shortcuts in Underspecified Visual Tasks

arXiv: https://arxiv.org/abs/2310.02230 · HTML: https://mlsanity.com/paper/2310.02230/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 25 references that are in the map.
- First posted on arXiv 3 Oct 2023.

## Builds on (most cited first, 21 of 21)

- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [Age Progression/Regression by Conditional Adversarial Autoencoder](https://mlsanity.com/paper/1702.08423/) (2017)
- [Confounding variables can degrade generalization performance of radiological deep learning models](https://mlsanity.com/paper/1807.00431/) (2018)
- [The Pitfalls of Simplicity Bias in Neural Networks](https://mlsanity.com/paper/2006.07710/) (2020)
- [Noise or Signal: The Role of Image Backgrounds in Object Recognition](https://mlsanity.com/paper/2006.09994/) (2020)
- [Diffusion Models already have a Semantic Latent Space](https://mlsanity.com/paper/2210.10960/) (2022)
- [Synthetic Data from Diffusion Models Improves ImageNet Classification](https://mlsanity.com/paper/2304.08466/) (2023)
- [Fake it till you make it: Learning transferable representations from synthetic ImageNet clones](https://mlsanity.com/paper/2212.08420/) (2022)
- [Uncovering the Disentanglement Capability in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2212.08698/) (2022)
- [Investigating Bias and Fairness in Facial Expression Recognition](https://mlsanity.com/paper/2007.10075/) (2020)
- [Evading the Simplicity Bias: Training a Diverse Set of Models Discovers Solutions with Superior OOD Generalization](https://mlsanity.com/paper/2105.05612/) (2021)
- [Agree to Disagree: Diversity through Disagreement for Better Transferability](https://mlsanity.com/paper/2202.04414/) (2022)
- [Which Shortcut Cues Will DNNs Choose? A Study from the Parameter-Space Perspective](https://mlsanity.com/paper/2110.03095/) (2021)
- [Diversify and Disambiguate: Learning From Underspecified Data](https://mlsanity.com/paper/2202.03418/) (2022)
- [Beyond Surface Statistics: Scene Representations in a Latent Diffusion Model](https://mlsanity.com/paper/2306.05720/) (2023)
- [Predicting is not Understanding: Recognizing and Addressing Underspecification in Machine Learning](https://mlsanity.com/paper/2207.02598/) (2022)
- [Ensembles of Locally Independent Prediction Models](https://mlsanity.com/paper/1911.01291/) (2019)
- [Learning Diverse Features in Vision Transformers for Improved Generalization](https://mlsanity.com/paper/2308.16274/) (2023)

## Built on by (most cited first, 4 of 4)

- [Shaping Inductive Bias in Diffusion Models through Frequency-Based Noise Control](https://mlsanity.com/paper/2502.10236/) (2025)
- [Mitigating Shortcut Learning with Diffusion Counterfactuals and Diverse Ensembles](https://mlsanity.com/paper/2311.16176/) (2023)
- [Identity-Consistent Video Generation under Large Facial-Angle Variations](https://mlsanity.com/paper/2603.21299/) (2026)
- [Learning What Matters: Steering Diffusion via Spectrally Anisotropic Forward Noise](https://mlsanity.com/paper/2510.09660/) (2025)

## Abstract (arXiv)

Spurious correlations in the data, where multiple cues are predictive of the target labels, often lead to shortcut learning phenomena, where a model may rely on erroneous, easy-to-learn, cues while ignoring reliable ones. In this work, we propose an ensemble diversification framework exploiting the generation of synthetic counterfactuals using Diffusion Probabilistic Models (DPMs). We discover that DPMs have the inherent capability to represent multiple visual cues independently, even when they are largely correlated in the training data. We leverage this characteristic to encourage model diversity and empirically show the efficacy of the approach with respect to several diversification objectives. We show that diffusion-guided diversification can lead models to avert attention from shortcut cues, achieving ensemble diversity performance comparable to previous methods requiring additional data collection.