# When Data Imbalance Helps: Robust Generalization Through Shortcut Saturation

arXiv: https://arxiv.org/abs/2607.10116 · HTML: https://mlsanity.com/paper/2607.10116/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 14 references that are in the map.
- First posted on arXiv 11 Jul 2026.

## Builds on (most cited first, 14 of 14)

- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [The Pitfalls of Simplicity Bias in Neural Networks](https://mlsanity.com/paper/2006.07710/) (2020)
- [An Investigation of Why Overparameterization Exacerbates Spurious Correlations](https://mlsanity.com/paper/2005.04345/) (2020)
- [Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit](https://mlsanity.com/paper/2207.08799/) (2022)
- [The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks](https://mlsanity.com/paper/2306.17844/) (2023)
- [On the Foundations of Shortcut Learning](https://mlsanity.com/paper/2310.16228/) (2023)
- [Domain Generalization via Gradient Surgery](https://mlsanity.com/paper/2108.01621/) (2021)
- [Class-Wise Difficulty-Balanced Loss for Solving Class-Imbalance](https://mlsanity.com/paper/2010.01824/) (2020)
- [Uncovering Memorization Effect in the Presence of Spurious Correlations](https://mlsanity.com/paper/2501.00961/) (2025)
- [Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wild](https://mlsanity.com/paper/2503.10065/) (2025)

## Abstract (arXiv)

We study robust generalization under spurious correlations: tasks where a shortcut feature is correlated with the true label in training but anti-correlated in an adversarial held-out split. Varying the spurious ratio r (the fraction of training examples where shortcut = true label) and model capacity, we find a counterintuitive result: data imbalance promotes generalization in sufficiently capable models. On a synthetic task where the true label is sum parity of an integer sequence and the shortcut is the parity of the maximum-valued element, a 2-layer, 2-head transformer generalized (reached 100% adversarial accuracy) in 0% of seeds at r=0.50 but 77% of seeds at r=0.90. The effect is absent in 1-layer models, where imbalance instead traps the model on the shortcut. Through mechanistic analysis -- gradient conflict dynamics, circuit evolution, and QK/OV circuit ablations -- we characterize a mechanistic pathway consistent with imbalance promoting generalization.