# Deep Bilevel Learning

arXiv: https://arxiv.org/abs/1809.01465 · HTML: https://mlsanity.com/paper/1809.01465/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Sep 2018.

## Built on by (most cited first, 30 of 49)

- [Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyond](https://mlsanity.com/paper/2101.11517/) (2021)
- [AutoBalance: Optimized Loss Functions for Imbalanced Data](https://mlsanity.com/paper/2201.01212/) (2022)
- [Lower Bounds and Accelerated Algorithms for Bilevel Optimization](https://mlsanity.com/paper/2102.03926/) (2021)
- [Enhanced Bilevel Optimization via Bregman Distance](https://mlsanity.com/paper/2107.12301/) (2021)
- [Transformers from an Optimization Perspective](https://mlsanity.com/paper/2205.13891/) (2022)
- [Sum of Ranked Range Loss for Supervised Learning](https://mlsanity.com/paper/2106.03300/) (2021)
- [Adaptive Sample Selection for Robust Learning under Label Noise](https://mlsanity.com/paper/2106.15292/) (2021)
- [An Optimization-Based Meta-Learning Model for MRI Reconstruction with Diverse Dataset](https://mlsanity.com/paper/2110.00715/) (2021)
- [A Generalized Doubly Robust Learning Framework for Debiasing Post-Click Conversion Rate Prediction](https://mlsanity.com/paper/2211.06684/) (2022)
- [Instance-Dependent Noisy Label Learning via Graphical Modelling](https://mlsanity.com/paper/2209.00906/) (2022)
- [Searching for Robustness: Loss Learning for Noisy Classification Tasks](https://mlsanity.com/paper/2103.00243/) (2021)
- [Can Less be More? When Increasing-to-Balancing Label Noise Rates Considered Beneficial](https://mlsanity.com/paper/2107.05913/) (2021)
- [Learning from data in the mixed adversarial non-adversarial case: Finding the helpers and ignoring the trolls](https://mlsanity.com/paper/2208.03295/) (2022)
- [Regroup Median Loss for Combating Label Noise](https://mlsanity.com/paper/2312.06273/) (2023)
- [Noisy Label Processing for Classification: A Survey](https://mlsanity.com/paper/2404.04159/) (2024)
- [Meta-Learned Invariant Risk Minimization](https://mlsanity.com/paper/2103.12947/) (2021)
- [Learning Sample Reweighting for Accuracy and Adversarial Robustness](https://mlsanity.com/paper/2210.11513/) (2022)
- [Probabilistic Machine Learning for Noisy Labels in Earth Observation](https://mlsanity.com/paper/2504.03478/) (2025)
- [MetaLabelNet: Learning to Generate Soft-Labels from Noisy-Labels](https://mlsanity.com/paper/2103.10869/) (2021)
- [DIVA: Dataset Derivative of a Learning Task](https://mlsanity.com/paper/2111.09785/) (2021)
- [Adaptive Mixing of Auxiliary Losses in Supervised Learning](https://mlsanity.com/paper/2202.03250/) (2022)
- [Multiplicative Reweighting for Robust Neural Network Optimization](https://mlsanity.com/paper/2102.12192/) (2021)
- [Label-noise-tolerant medical image classification via self-attention and self-supervised learning](https://mlsanity.com/paper/2306.09718/) (2023)
- [Batch Inverse-Variance Weighting: Deep Heteroscedastic Regression](https://mlsanity.com/paper/2107.04497/) (2021)
- [The Dynamic of Consensus in Deep Networks and the Identification of Noisy Labels](https://mlsanity.com/paper/2210.00583/) (2022)
- [GenKL: An Iterative Framework for Resolving Label Ambiguity and Label Non-conformity in Web Images Via a New Generalized KL Divergence](https://mlsanity.com/paper/2307.09810/) (2023)
- [UCE-FID: Using Large Unlabeled, Medium Crowdsourced-Labeled, and Small Expert-Labeled Tweets for Foodborne Illness Detection](https://mlsanity.com/paper/2312.01225/) (2023)
- [Understanding and Mitigating Human-Labelling Errors in Supervised Contrastive Learning](https://mlsanity.com/paper/2403.06289/) (2024)
- [Training Classifiers that are Universally Robust to All Label Noise Levels](https://mlsanity.com/paper/2105.13892/) (2021)
- [Fair Machine Learning under Limited Demographically Labeled Data](https://mlsanity.com/paper/2106.04757/) (2021)

## Abstract (arXiv)

We present a novel regularization approach to train neural networks that enjoys better generalization and test error than standard stochastic gradient descent. Our approach is based on the principles of cross-validation, where a validation set is used to limit the model overfitting. We formulate such principles as a bilevel optimization problem. This formulation allows us to define the optimization of a cost on the validation set subject to another optimization on the training set. The overfitting is controlled by introducing weights on each mini-batch in the training set and by choosing their values so that they minimize the error on the validation set. In practice, these weights define mini-batch learning rates in a gradient descent update equation that favor gradients with better generalization capabilities. Because of its simplicity, this approach can be integrated with other regularization methods and training schemes. We evaluate extensively our proposed algorithm on several neural network architectures and datasets, and find that it consistently improves the generalization of the model, especially when labels are noisy.