# IMAE for Noise-Robust Learning: Mean Absolute Error Does Not Treat Examples Equally and Gradient Magnitude's Variance Matters

arXiv: https://arxiv.org/abs/1903.12141 · HTML: https://mlsanity.com/paper/1903.12141/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Mar 2019.

## Built on by (most cited first, 30 of 33)

- [Investigating Why Contrastive Learning Benefits Robustness Against Label Noise](https://mlsanity.com/paper/2201.12498/) (2022)
- [LongReMix: Robust Learning with High Confidence Samples in a Noisy Label Environment](https://mlsanity.com/paper/2103.04173/) (2021)
- [Uncertainty Estimation via Response Scaling for Pseudo-mask Noise Mitigation in Weakly-supervised Semantic Segmentation](https://mlsanity.com/paper/2112.07431/) (2021)
- [Twin Contrastive Learning with Noisy Labels](https://mlsanity.com/paper/2303.06930/) (2023)
- [PropMix: Hard Sample Filtering and Proportional MixUp for Learning with Noisy Labels](https://mlsanity.com/paper/2110.11809/) (2021)
- [Instance-Dependent Noisy Label Learning via Graphical Modelling](https://mlsanity.com/paper/2209.00906/) (2022)
- [Improving Medical Image Classification with Label Noise Using Dual-uncertainty Estimation](https://mlsanity.com/paper/2103.00528/) (2021)
- [ScanMix: Learning from Severe Label Noise via Semantic Clustering and Semi-Supervised Learning](https://mlsanity.com/paper/2103.11395/) (2021)
- [Not All Knowledge Is Created Equal: Mutual Distillation of Confident Knowledge](https://mlsanity.com/paper/2106.01489/) (2021)
- [Unlocking the Power of Open Set : A New Perspective for Open-Set Noisy Label Learning](https://mlsanity.com/paper/2305.04203/) (2023)
- [PARS: Pseudo-Label Aware Robust Sample Selection for Learning with Noisy Labels](https://mlsanity.com/paper/2201.10836/) (2022)
- [Robust Classification from Noisy Labels: Integrating Additional Knowledge for Chest Radiography Abnormality Assessment](https://mlsanity.com/paper/2104.05261/) (2021)
- [Confidence Adaptive Regularization for Deep Learning with Noisy Labels](https://mlsanity.com/paper/2108.08212/) (2021)
- [ProSelfLC: Progressive Self Label Correction Towards A Low-Temperature Entropy State](https://mlsanity.com/paper/2207.00118/) (2022)
- [Elucidating Robust Learning with Uncertainty-Aware Corruption Pattern Estimation](https://mlsanity.com/paper/2111.01632/) (2021)
- [Tripartite: Tackle Noisy Labels by a More Precise Partition](https://mlsanity.com/paper/2202.09579/) (2022)
- [A Study on the Impact of Data Augmentation for Training Convolutional Neural Networks in the Presence of Noisy Labels](https://mlsanity.com/paper/2208.11176/) (2022)
- [A Novel Technique for Robust Training of Deep Networks With Multisource Weak Labeled Remote Sensing Data](https://mlsanity.com/paper/2510.05760/) (2025)
- [Affect Expression Behaviour Analysis in the Wild using Consensual Collaborative Training](https://mlsanity.com/paper/2107.05736/) (2021)
- [Dynamic Adaptive Threshold based Learning for Noisy Annotations Robust Facial Expression Recognition](https://mlsanity.com/paper/2208.10221/) (2022)
- [Learning from Noisy Labels with Coarse-to-Fine Sample Credibility Modeling](https://mlsanity.com/paper/2208.10683/) (2022)
- [Maximising the Utility of Validation Sets for Imbalanced Noisy-label Meta-learning](https://mlsanity.com/paper/2208.08132/) (2022)
- [Evaluating the Impact of Loss Function Variation in Deep Learning for Classification](https://mlsanity.com/paper/2210.16003/) (2022)
- [Noisy Early Stopping for Noisy Labels](https://mlsanity.com/paper/2409.06830/) (2024)
- [Generation and Analysis of Feature-Dependent Pseudo Noise for Training Deep Neural Networks](https://mlsanity.com/paper/2105.10796/) (2021)
- [Consensual Collaborative Training And Knowledge Distillation Based Facial Expression Recognition Under Noisy Annotations](https://mlsanity.com/paper/2107.04746/) (2021)
- [Joint Attention-Driven Domain Fusion and Noise-Tolerant Learning for Multi-Source Domain Adaptation](https://mlsanity.com/paper/2208.02947/) (2022)
- [Establishment of Neural Networks Robust to Label Noise](https://mlsanity.com/paper/2211.15279/) (2022)
- [A Curriculum View of Robust Loss Functions](https://mlsanity.com/paper/2305.02139/) (2023)
- [Robust Feature Learning Against Noisy Labels](https://mlsanity.com/paper/2307.04312/) (2023)

## Abstract (arXiv)

In this work, we study robust deep learning against abnormal training data from the perspective of example weighting built in empirical loss functions, i.e., gradient magnitude with respect to logits, an angle that is not thoroughly studied so far. Consequently, we have two key findings: (1) Mean Absolute Error (MAE) Does Not Treat Examples Equally. We present new observations and insightful analysis about MAE, which is theoretically proved to be noise-robust. First, we reveal its underfitting problem in practice. Second, we analyse that MAE's noise-robustness is from emphasising on uncertain examples instead of treating training samples equally, as claimed in prior work. (2) The Variance of Gradient Magnitude Matters. We propose an effective and simple solution to enhance MAE's fitting ability while preserving its noise-robustness. Without changing MAE's overall weighting scheme, i.e., what examples get higher weights, we simply change its weighting variance non-linearly so that the impact ratio between two examples are adjusted. Our solution is termed Improved MAE (IMAE). We prove IMAE's effectiveness using extensive experiments: image classification under clean labels, synthetic label noise, and real-world unknown noise.