# Faster Meta Update Strategy for Noise-Robust Deep Learning

arXiv: https://arxiv.org/abs/2104.15092 · HTML: https://mlsanity.com/paper/2104.15092/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 64 references that are in the map.
- First posted on arXiv 30 Apr 2021.

## Builds on (most cited first, 30 of 47)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [Large-Scale Long-Tailed Recognition in an Open World](https://mlsanity.com/paper/1904.05160/) (2019)
- [DivideMix: Learning with Noisy Labels as Semi-supervised Learning](https://mlsanity.com/paper/2002.07394/) (2020)
- [Learning to Reweight Examples for Robust Deep Learning](https://mlsanity.com/paper/1803.09050/) (2018)
- [Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach](https://mlsanity.com/paper/1609.03683/) (2016)
- [MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels](https://mlsanity.com/paper/1712.05055/) (2017)
- [Exploring the Limits of Weakly Supervised Pretraining](https://mlsanity.com/paper/1805.00932/) (2018)
- [Confident Learning: Estimating Uncertainty in Dataset Labels](https://mlsanity.com/paper/1911.00068/) (2019)
- [BBN: Bilateral-Branch Network with Cumulative Learning for Long-Tailed Visual Recognition](https://mlsanity.com/paper/1912.02413/) (2019)
- [How does Disagreement Help Generalization against Label Corruption?](https://mlsanity.com/paper/1901.04215/) (2019)
- [Unsupervised Label Noise Modeling and Loss Correction](https://mlsanity.com/paper/1904.11238/) (2019)
- [Joint Optimization Framework for Learning with Noisy Labels](https://mlsanity.com/paper/1803.11364/) (2018)
- [Combating noisy labels by agreement: A joint training method with co-regularization](https://mlsanity.com/paper/2003.02752/) (2020)
- [WebVision Database: Visual Learning and Understanding from Web Data](https://mlsanity.com/paper/1708.02862/) (2017)
- [Identifying Mislabeled Data using the Area Under the Margin Ranking](https://mlsanity.com/paper/2001.10528/) (2020)
- [Decoupling "when to update" from "how to update"](https://mlsanity.com/paper/1706.02613/) (2017)

## Built on by (most cited first, 30 of 38)

- [Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling](https://mlsanity.com/paper/2111.12698/) (2021)
- [Learngene: Inheriting Condensed Knowledge from the Ancestry Model to Descendant Models](https://mlsanity.com/paper/2305.02279/) (2023)
- [Learning from Noisy Labels with Decoupled Meta Label Purifier](https://mlsanity.com/paper/2302.06810/) (2023)
- [PropMix: Hard Sample Filtering and Proportional MixUp for Learning with Noisy Labels](https://mlsanity.com/paper/2110.11809/) (2021)
- [Instance-Dependent Noisy Label Learning via Graphical Modelling](https://mlsanity.com/paper/2209.00906/) (2022)
- [How to Sift Out a Clean Data Subset in the Presence of Data Poisoning?](https://mlsanity.com/paper/2210.06516/) (2022)
- [Delving into Sample Loss Curve to Embrace Noisy and Imbalanced Data](https://mlsanity.com/paper/2201.00849/) (2021)
- [CLIPCleaner: Cleaning Noisy Labels with CLIP](https://mlsanity.com/paper/2408.10012/) (2024)
- [SSR: An Efficient and Robust Framework for Learning with Unknown Label Noise](https://mlsanity.com/paper/2111.11288/) (2021)
- [Self-supervised and Supervised Joint Training for Resource-rich Machine Translation](https://mlsanity.com/paper/2106.04060/) (2021)
- [Is your noise correction noisy? PLS: Robustness to label noise with two stage detection](https://mlsanity.com/paper/2210.04578/) (2022)
- [L2B: Learning to Bootstrap Robust Models for Combating Label Noise](https://mlsanity.com/paper/2202.04291/) (2022)
- [Manifold DivideMix: A Semi-Supervised Contrastive Learning Framework for Severe Label Noise](https://mlsanity.com/paper/2308.06861/) (2023)
- [Coupled Confusion Correction: Learning from Crowds with Sparse Annotations](https://mlsanity.com/paper/2312.07331/) (2023)
- [Learning to Complement and to Defer to Multiple Users](https://mlsanity.com/paper/2407.07003/) (2024)
- [Embedding contrastive unsupervised features to cluster in- and out-of-distribution noise in corrupted image datasets](https://mlsanity.com/paper/2207.01573/) (2022)
- [Dynamic Loss For Robust Learning](https://mlsanity.com/paper/2211.12506/) (2022)
- [Enhanced Meta Label Correction for Coping with Label Corruption](https://mlsanity.com/paper/2305.12961/) (2023)
- [Combating Label Noise With A General Surrogate Model For Sample Selection](https://mlsanity.com/paper/2310.10463/) (2023)
- [Learning to Complement with Multiple Humans](https://mlsanity.com/paper/2311.13172/) (2023)
- [Instance-dependent Noisy-label Learning with Graphical Model Based Noise-rate Estimation](https://mlsanity.com/paper/2305.19486/) (2023)
- [Curvature Learning for Generalization of Hyperbolic Neural Networks](https://mlsanity.com/paper/2508.17232/) (2025)
- [Contrastive Knowledge Transfer and Robust Optimization for Secure Alignment of Large Language Models](https://mlsanity.com/paper/2510.27077/) (2025)
- [Online Meta-Learning for Model Update Aggregation in Federated Learning for Click-Through Rate Prediction](https://mlsanity.com/paper/2209.00629/) (2022)
- [MetaSeg: Content-Aware Meta-Net for Omni-Supervised Semantic Segmentation](https://mlsanity.com/paper/2401.11738/) (2024)
- [Maximising the Utility of Validation Sets for Imbalanced Noisy-label Meta-learning](https://mlsanity.com/paper/2208.08132/) (2022)
- [Asymmetric Co-teaching with Multi-view Consensus for Noisy Label Learning](https://mlsanity.com/paper/2301.01143/) (2023)
- [Towards the Identifiability in Noisy Label Learning: A Multinomial Mixture Modelling Approach](https://mlsanity.com/paper/2301.01405/) (2023)
- [PASS: Peer-Agreement based Sample Selection for training with Noisy Labels](https://mlsanity.com/paper/2303.10802/) (2023)
- [Self-Distillation with Meta Learning for Knowledge Graph Completion](https://mlsanity.com/paper/2305.12209/) (2023)

## Abstract (arXiv)

It has been shown that deep neural networks are prone to overfitting on biased training data. Towards addressing this issue, meta-learning employs a meta model for correcting the training bias. Despite the promising performances, super slow training is currently the bottleneck in the meta learning approaches. In this paper, we introduce a novel Faster Meta Update Strategy (FaMUS) to replace the most expensive step in the meta gradient computation with a faster layer-wise approximation. We empirically find that FaMUS yields not only a reasonably accurate but also a low-variance approximation of the meta gradient. We conduct extensive experiments to verify the proposed method on two tasks. We show our method is able to save two-thirds of the training time while still maintaining the comparable or achieving even better generalization performance. In particular, our method achieves the state-of-the-art performance on both synthetic and realistic noisy labels, and obtains promising performance on long-tailed recognition on standard benchmarks.