# Model Agnostic Sample Reweighting for Out-of-Distribution Learning

arXiv: https://arxiv.org/abs/2301.09819 · HTML: https://mlsanity.com/paper/2301.09819/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 74 of its 100 references that are in the map.
- First posted on arXiv 24 Jan 2023.

## Builds on (most cited first, 30 of 74)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [Out-of-Distribution Generalization via Risk Extrapolation (REx)](https://mlsanity.com/paper/2003.00688/) (2020)
- [Learning to Reweight Examples for Robust Deep Learning](https://mlsanity.com/paper/1803.09050/) (2018)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [Learning Sparse Neural Networks through L0 Regularization](https://mlsanity.com/paper/1712.01312/) (2017)
- [Causal inference using invariant prediction: identification and confidence intervals](https://mlsanity.com/paper/1501.01332/) (2015)
- [Just Train Twice: Improving Group Robustness without Training Group Information](https://mlsanity.com/paper/2107.09044/) (2021)
- [Towards Out-Of-Distribution Generalization: A Survey](https://mlsanity.com/paper/2108.13624/) (2021)
- [Gradient-based Hyperparameter Optimization through Reversible Learning](https://mlsanity.com/paper/1502.03492/) (2015)
- [The Pitfalls of Simplicity Bias in Neural Networks](https://mlsanity.com/paper/2006.07710/) (2020)
- [Learning Models with Uniform Performance via Distributionally Robust Optimization](https://mlsanity.com/paper/1810.08750/) (2018)
- [An Investigation of Why Overparameterization Exacerbates Spurious Correlations](https://mlsanity.com/paper/2005.04345/) (2020)
- [Optimizing Millions of Hyperparameters by Implicit Differentiation](https://mlsanity.com/paper/1911.02590/) (2019)
- [PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search](https://mlsanity.com/paper/1907.05737/) (2019)
- [The Risks of Invariant Risk Minimization](https://mlsanity.com/paper/2010.05761/) (2020)
- [Hyperparameter optimization with approximate gradient](https://mlsanity.com/paper/1602.02355/) (2016)
- [Recurrent Independent Mechanisms](https://mlsanity.com/paper/1909.10893/) (2019)
- [Coresets via Bilevel Optimization for Continual Learning and Streaming](https://mlsanity.com/paper/2006.03875/) (2020)
- [Truncated Back-propagation for Bilevel Optimization](https://mlsanity.com/paper/1810.10667/) (2018)

## Built on by (most cited first, 30 of 44)

- [Mitigating the Alignment Tax of RLHF](https://mlsanity.com/paper/2309.06256/) (2023)
- [Image Textualization: An Automatic Framework for Creating Accurate and Detailed Image Descriptions](https://mlsanity.com/paper/2406.07502/) (2024)
- [A Sober Look at the Robustness of CLIPs to Spurious Features](https://mlsanity.com/paper/2403.11497/) (2024)
- [ScaleBiO: Scalable Bilevel Optimization for LLM Data Reweighting](https://mlsanity.com/paper/2406.19976/) (2024)
- [What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?](https://mlsanity.com/paper/2305.18882/) (2023)
- [Understanding and Improving Feature Learning for Out-of-Distribution Generalization](https://mlsanity.com/paper/2304.11327/) (2023)
- [Spurious Feature Diversification Improves Out-of-distribution Generalization](https://mlsanity.com/paper/2309.17230/) (2023)
- [FACTS: First Amplify Correlations and Then Slice to Discover Bias](https://mlsanity.com/paper/2309.17430/) (2023)
- [Rethinking the Evaluation Protocol of Domain Generalization](https://mlsanity.com/paper/2305.15253/) (2023)
- [Near-Optimal Nonconvex-Strongly-Convex Bilevel Optimization with Fully First-Order Oracles](https://mlsanity.com/paper/2306.14853/) (2023)
- [A Textbook Remedy for Domain Shifts: Knowledge Priors for Medical Image Analysis](https://mlsanity.com/paper/2405.14839/) (2024)
- [Unlocking the Power of Open Set : A New Perspective for Open-Set Noisy Label Learning](https://mlsanity.com/paper/2305.04203/) (2023)
- [Understanding Domain Generalization: A Noise Robustness Perspective](https://mlsanity.com/paper/2401.14846/) (2024)
- [Distributionally Generative Augmentation for Fair Facial Attribute Classification](https://mlsanity.com/paper/2403.06606/) (2024)
- [Group-robust Sample Reweighting for Subpopulation Shifts via Influence Functions](https://mlsanity.com/paper/2503.07315/) (2025)
- [Improving Generalization via Meta-Learning on Hard Samples](https://mlsanity.com/paper/2403.12236/) (2024)
- [AIM-Fair: Advancing Algorithmic Fairness via Selectively Fine-Tuning Biased Models with Contextual Synthetic Data](https://mlsanity.com/paper/2503.05665/) (2025)
- [Quantitatively Measuring and Contrastively Exploring Heterogeneity for Domain Generalization](https://mlsanity.com/paper/2305.15889/) (2023)
- [Optimal Sample Selection Through Uncertainty Estimation and Its Application in Deep Learning](https://mlsanity.com/paper/2309.02476/) (2023)
- [ReFine: Boosting Time Series Prediction of Extreme Events by Reweighting and Fine-tuning](https://mlsanity.com/paper/2409.14232/) (2024)
- [Mitigating Spurious Correlations via Disagreement Probability](https://mlsanity.com/paper/2411.01757/) (2024)
- [Computational Budget Should Be Considered in Data Selection](https://mlsanity.com/paper/2510.16806/) (2025)
- [Predictive Heterogeneity: Measures and Applications](https://mlsanity.com/paper/2304.00305/) (2023)
- [Boosting Fair Classifier Generalization through Adaptive Priority Reweighing](https://mlsanity.com/paper/2309.08375/) (2023)
- [Mixture Data for Training Cannot Ensure Out-of-distribution Generalization](https://mlsanity.com/paper/2312.16243/) (2023)
- [Soft-Label Integration for Robust Toxicity Classification](https://mlsanity.com/paper/2410.14894/) (2024)
- [DivIL: Unveiling and Addressing Over-Invariance for Out-of- Distribution Generalization](https://mlsanity.com/paper/2502.12413/) (2025)
- [Class-Conditional Distribution Balancing for Group Robust Classification](https://mlsanity.com/paper/2504.17314/) (2025)
- [Fine-Grained Class-Conditional Distribution Balancing for Debiased Learning](https://mlsanity.com/paper/2505.06831/) (2025)
- [KAIROS: Scalable Model-Agnostic Data Valuation](https://mlsanity.com/paper/2506.23799/) (2025)

## Abstract (arXiv)

Distributionally robust optimization (DRO) and invariant risk minimization (IRM) are two popular methods proposed to improve out-of-distribution (OOD) generalization performance of machine learning models. While effective for small models, it has been observed that these methods can be vulnerable to overfitting with large overparameterized models. This work proposes a principled method, Model Agnostic samPLe rEweighting (MAPLE), to effectively address OOD problem, especially in overparameterized scenarios. Our key idea is to find an effective reweighting of the training samples so that the standard empirical risk minimization training of a large model on the weighted training data leads to superior OOD generalization performance. The overfitting issue is addressed by considering a bilevel formulation to search for the sample reweighting, in which the generalization complexity depends on the search space of sample weights instead of the model size. We present theoretical analysis in linear case to prove the insensitivity of MAPLE to model size, and empirically verify its superiority in surpassing state-of-the-art methods by a large margin. Code is available at urlhttps://github.com/x-zho14/MAPLE.