# Improving Generalization via Meta-Learning on Hard Samples

arXiv: https://arxiv.org/abs/2403.12236 · HTML: https://mlsanity.com/paper/2403.12236/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 43 references that are in the map.
- First posted on arXiv 18 Mar 2024.

## Builds on (most cited first, 28 of 28)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [Meta-Learning in Neural Networks: A Survey](https://mlsanity.com/paper/2004.05439/) (2020)
- [Learning to Reweight Examples for Robust Deep Learning](https://mlsanity.com/paper/1803.09050/) (2018)
- [Just Train Twice: Improving Group Robustness without Training Group Information](https://mlsanity.com/paper/2107.09044/) (2021)
- [Agnostic Federated Learning](https://mlsanity.com/paper/1902.00146/) (2019)
- [Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML](https://mlsanity.com/paper/1909.09157/) (2019)
- [Learning Models with Uniform Performance via Distributionally Robust Optimization](https://mlsanity.com/paper/1810.08750/) (2018)
- [Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt](https://mlsanity.com/paper/2206.07137/) (2022)
- [Variance-based regularization with convex objectives](https://mlsanity.com/paper/1610.02581/) (2016)
- [Deep Stable Learning for Out-Of-Distribution Generalization](https://mlsanity.com/paper/2104.07876/) (2021)
- [The iWildCam 2020 Competition Dataset](https://mlsanity.com/paper/2004.10340/) (2020)
- [Learning Fast Sample Re-weighting Without Reward Data](https://mlsanity.com/paper/2109.03216/) (2021)
- [Probabilistic Margins for Instance Reweighting in Adversarial Training](https://mlsanity.com/paper/2106.07904/) (2021)
- [DORO: Distributional and Outlier Robust Optimization](https://mlsanity.com/paper/2106.06142/) (2021)
- [Model Agnostic Sample Reweighting for Out-of-Distribution Learning](https://mlsanity.com/paper/2301.09819/) (2023)
- [Difficulty-aware Meta-learning for Rare Disease Diagnosis](https://mlsanity.com/paper/1907.00354/) (2019)
- [Learning to Split for Automatic Bias Detection](https://mlsanity.com/paper/2204.13749/) (2022)
- [Generalized Data Weighting via Class-level Gradient Manipulation](https://mlsanity.com/paper/2111.00056/) (2021)
- [Fast Context Adaptation via Meta-Learning](https://mlsanity.com/paper/1810.03642/) (2018)
- [Learning Sample Reweighting for Accuracy and Adversarial Robustness](https://mlsanity.com/paper/2210.11513/) (2022)
- [Accelerated Proximal Alternating Gradient-Descent-Ascent for Nonconvex Minimax Machine Learning](https://mlsanity.com/paper/2112.11663/) (2021)
- [Instance-Conditional Timescales of Decay for Non-Stationary Learning](https://mlsanity.com/paper/2212.05908/) (2022)
- [Mix and Match: An Optimistic Tree-Search Approach for Learning Models from Mixture Distributions](https://mlsanity.com/paper/1907.10154/) (2019)

## Built on by (most cited first, 5 of 5)

- [Uncertainty-Guided Checkpoint Selection for Reinforcement Finetuning of Large Language Models](https://mlsanity.com/paper/2511.09864/) (2025)
- [Model-Agnostic Meta Learning for Class Imbalance Adaptation](https://mlsanity.com/paper/2604.18759/) (2026)
- [LiLAW: Lightweight Learnable Adaptive Weighting to Learn Sample Difficulty & Improve Noisy Training](https://mlsanity.com/paper/2509.20786/) (2025)
- [Data-Efficient Training by Evolved Sampling](https://mlsanity.com/paper/2509.23461/) (2025)
- [Humanoid-inspired Causal Representation Learning for Domain Generalization](https://mlsanity.com/paper/2510.16382/) (2025)

## Abstract (arXiv)

Learned reweighting (LRW) approaches to supervised learning use an optimization criterion to assign weights for training instances, in order to maximize performance on a representative validation dataset. We pose and formalize the problem of optimized selection of the validation set used in LRW training, to improve classifier generalization. In particular, we show that using hard-to-classify instances in the validation set has both a theoretical connection to, and strong empirical evidence of generalization. We provide an efficient algorithm for training this meta-optimized model, as well as a simple train-twice heuristic for careful comparative study. We demonstrate that LRW with easy validation data performs consistently worse than LRW with hard validation data, establishing the validity of our meta-optimization problem. Our proposed algorithm outperforms a wide range of baselines on a range of datasets and domain shift challenges (Imagenet-1K, CIFAR-100, Clothing-1M, CAMELYON, WILDS, etc.), with ~1% gains using VIT-B on Imagenet. We also show that using naturally hard examples for validation (Imagenet-R / Imagenet-A) in LRW training for Imagenet improves performance on both clean and naturally hard test instances by 1-2%. Secondary analyses show that using hard validation data in an LRW framework improves margins on test data, hinting at the mechanism underlying our empirical gains. We believe this work opens up new research directions for the meta-optimization of meta-learning in a supervised learning context.