# Robust Learning with Progressive Data Expansion Against Spurious Correlation

arXiv: https://arxiv.org/abs/2306.04949 · HTML: https://mlsanity.com/paper/2306.04949/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 51 references that are in the map.
- First posted on arXiv 8 Jun 2023.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [Just Train Twice: Improving Group Robustness without Training Group Information](https://mlsanity.com/paper/2107.09044/) (2021)
- [Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations](https://mlsanity.com/paper/2204.02937/) (2022)
- [An Investigation of Why Overparameterization Exacerbates Spurious Correlations](https://mlsanity.com/paper/2005.04345/) (2020)
- [Environment Inference for Invariant Learning](https://mlsanity.com/paper/2010.07249/) (2020)
- [Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning](https://mlsanity.com/paper/2012.09816/) (2020)
- [What is the Effect of Importance Weighting in Deep Learning?](https://mlsanity.com/paper/1812.03372/) (2018)
- [No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problems](https://mlsanity.com/paper/2011.12945/) (2020)
- [Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlations](https://mlsanity.com/paper/2203.01517/) (2022)
- [Variance-based regularization with convex objectives](https://mlsanity.com/paper/1610.02581/) (2016)
- [On Feature Learning in the Presence of Spurious Correlations](https://mlsanity.com/paper/2210.11369/) (2022)
- [Distributionally Robust Language Modeling](https://mlsanity.com/paper/1909.02060/) (2019)
- [Change is Hard: A Closer Look at Subpopulation Shift](https://mlsanity.com/paper/2302.12254/) (2023)
- [Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learning](https://mlsanity.com/paper/2105.15134/) (2021)
- [Benign Overfitting in Two-layer Convolutional Neural Networks](https://mlsanity.com/paper/2202.06526/) (2022)
- [Model Patching: Closing the Subgroup Performance Gap with Data Augmentation](https://mlsanity.com/paper/2008.06775/) (2020)
- [Coping with Label Shift via Distributionally Robust Optimisation](https://mlsanity.com/paper/2010.12230/) (2020)
- [Self-training Avoids Using Spurious Features Under Domain Shift](https://mlsanity.com/paper/2006.10032/) (2020)
- [Heteroskedastic and Imbalanced Deep Learning with Adaptive Regularization](https://mlsanity.com/paper/2006.15766/) (2020)
- [Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularization](https://mlsanity.com/paper/2108.11371/) (2021)
- [Mitigating Spurious Correlations in Multi-modal Models during Fine-tuning](https://mlsanity.com/paper/2304.03916/) (2023)
- [Identifying Spurious Biases Early in Training through the Lens of Simplicity Bias](https://mlsanity.com/paper/2305.18761/) (2023)
- [Towards understanding how momentum improves generalization in deep learning](https://mlsanity.com/paper/2207.05931/) (2022)

## Built on by (most cited first, 30 of 39)

- [SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Models](https://mlsanity.com/paper/2403.07384/) (2024)
- [Does Invariant Graph Learning via Environment Augmentation Learn Invariance?](https://mlsanity.com/paper/2310.19035/) (2023)
- [Prompt-CAM: Making Vision Transformers Interpretable for Fine-Grained Analysis](https://mlsanity.com/paper/2501.09333/) (2025)
- [Controllable Prompt Tuning For Balancing Group Distributional Robustness](https://mlsanity.com/paper/2403.02695/) (2024)
- [Learning Robust Classifiers with Self-Guided Spurious Correlation Mitigation](https://mlsanity.com/paper/2405.03649/) (2024)
- [Navigating the Shortcut Maze: A Comprehensive Analysis of Shortcut Learning in Text Classification by Language Models](https://mlsanity.com/paper/2409.17455/) (2024)
- [A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Future](https://mlsanity.com/paper/2412.14056/) (2024)
- [Enhancing Domain Adaptation through Prompt Gradient Alignment](https://mlsanity.com/paper/2406.09353/) (2024)
- [Mitigating Spurious Correlation via Distributionally Robust Learning with Hierarchical Ambiguity Sets](https://mlsanity.com/paper/2510.02818/) (2025)
- [Mitigating Simplicity Bias in Deep Learning for Improved OOD Generalization and Robustness](https://mlsanity.com/paper/2310.06161/) (2023)
- [DASH: Warm-Starting Neural Network Training in Stationary Settings without Loss of Plasticity](https://mlsanity.com/paper/2410.23495/) (2024)
- [Severing Spurious Correlations with Data Pruning](https://mlsanity.com/paper/2503.18258/) (2025)
- [How does promoting the minority fraction affect generalization? A theoretical study of the one-hidden-layer neural network on group imbalance](https://mlsanity.com/paper/2403.07310/) (2024)
- [Group-robust Sample Reweighting for Subpopulation Shifts via Influence Functions](https://mlsanity.com/paper/2503.07315/) (2025)
- [Provably Neural Active Learning Succeeds via Prioritizing Perplexing Samples](https://mlsanity.com/paper/2406.03944/) (2024)
- [SLIM: Spuriousness Mitigation with Minimal Human Annotations](https://mlsanity.com/paper/2407.05594/) (2024)
- [Trustworthy AI: Safety, Bias, and Privacy -- A Survey](https://mlsanity.com/paper/2502.10450/) (2025)
- [ShortcutProbe: Probing Prediction Shortcuts for Learning Robust Models](https://mlsanity.com/paper/2505.13910/) (2025)
- [Diverse Prototypical Ensembles Improve Robustness to Subpopulation Shift](https://mlsanity.com/paper/2505.23027/) (2025)
- [Challenges and Opportunities in Improving Worst-Group Generalization in Presence of Spurious Features](https://mlsanity.com/paper/2306.11957/) (2023)
- [Multi-View Incongruity Learning for Multimodal Sarcasm Detection](https://mlsanity.com/paper/2412.00756/) (2024)
- [Distributionally Robust Classification for Multi-source Unsupervised Domain Adaptation](https://mlsanity.com/paper/2601.21315/) (2026)
- [Changing the Training Data Distribution to Reduce Simplicity Bias Improves In-distribution Generalization](https://mlsanity.com/paper/2404.17768/) (2024)
- [Do We Need All the Synthetic Data? Targeted Image Augmentation via Diffusion Models](https://mlsanity.com/paper/2505.21574/) (2025)
- [NeuronTune: Towards Self-Guided Spurious Bias Mitigation](https://mlsanity.com/paper/2505.24048/) (2025)
- [Superclass-Guided Representation Disentanglement for Spurious Correlation Mitigation](https://mlsanity.com/paper/2508.08570/) (2025)
- [Does Weak-to-strong Generalization Happen under Spurious Correlations?](https://mlsanity.com/paper/2509.24005/) (2025)
- [Spurious Correlation-Aware Embedding Regularization for Worst-Group Robustness](https://mlsanity.com/paper/2511.04401/) (2025)
- [Revisiting Spurious Correlation in Domain Generalization](https://mlsanity.com/paper/2406.11517/) (2024)
- [Single-weight Model Editing for Post-hoc Spurious Correlation Neutralization](https://mlsanity.com/paper/2501.14182/) (2025)

## Abstract (arXiv)

While deep learning models have shown remarkable performance in various tasks, they are susceptible to learning non-generalizable spurious features rather than the core features that are genuinely correlated to the true label. In this paper, beyond existing analyses of linear models, we theoretically examine the learning process of a two-layer nonlinear convolutional neural network in the presence of spurious features. Our analysis suggests that imbalanced data groups and easily learnable spurious features can lead to the dominance of spurious features during the learning process. In light of this, we propose a new training algorithm called PDE that efficiently enhances the model's robustness for a better worst-group performance. PDE begins with a group-balanced subset of training data and progressively expands it to facilitate the learning of the core features. Experiments on synthetic and real-world benchmark datasets confirm the superior performance of our method on models such as ResNets and Transformers. On average, our method achieves a 2.8% improvement in worst-group accuracy compared with the state-of-the-art method, while enjoying up to 10x faster training efficiency. Codes are available at https://github.com/uclaml/PDE.