# How Does a Neural Network's Architecture Impact Its Robustness to Noisy Labels?

arXiv: https://arxiv.org/abs/2012.12896 · HTML: https://mlsanity.com/paper/2012.12896/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Dec 2020.

## Built on by (most cited first, 12 of 12)

- [Sparse Mixture-of-Experts are Domain Generalizable Learners](https://mlsanity.com/paper/2206.04046/) (2022)
- [Detecting Corrupted Labels Without Training a Model to Predict](https://mlsanity.com/paper/2110.06283/) (2021)
- [Graph Neural Networks are Dynamic Programmers](https://mlsanity.com/paper/2203.15544/) (2022)
- [CTRL: Clustering Training Losses for Label Error Detection](https://mlsanity.com/paper/2208.08464/) (2022)
- [DeTiME: Diffusion-Enhanced Topic Modeling using Encoder-decoder based LLM](https://mlsanity.com/paper/2310.15296/) (2023)
- [Neighborhood-Regularized Self-Training for Learning with Few Labels](https://mlsanity.com/paper/2301.03726/) (2023)
- [Strong inductive biases provably prevent harmless interpolation](https://mlsanity.com/paper/2301.07605/) (2023)
- [Quantifying and mitigating the impact of label errors on model disparity metrics](https://mlsanity.com/paper/2310.02533/) (2023)
- [Stochastic Resetting Mitigates Latent Gradient Bias of SGD from Label Noise](https://mlsanity.com/paper/2406.00396/) (2024)
- [Leveraging Unlabeled Data to Track Memorization](https://mlsanity.com/paper/2212.04461/) (2022)
- [Revisiting Early-Learning Regularization When Federated Learning Meets Noisy Labels](https://mlsanity.com/paper/2402.05353/) (2024)
- [Neural Architecture Search of Sample Reweighting Networks for Complex Distribution Shift](https://mlsanity.com/paper/2606.22991/) (2026)

## Abstract (arXiv)

Noisy labels are inevitable in large real-world datasets. In this work, we explore an area understudied by previous works -- how the network's architecture impacts its robustness to noisy labels. We provide a formal framework connecting the robustness of a network to the alignments between its architecture and target/noise functions. Our framework measures a network's robustness via the predictive power in its representations -- the test performance of a linear model trained on the learned representations using a small set of clean labels. We hypothesize that a network is more robust to noisy labels if its architecture is more aligned with the target function than the noise. To support our hypothesis, we provide both theoretical and empirical evidence across various neural network architectures and different domains. We also find that when the network is well-aligned with the target function, its predictive power in representations could improve upon state-of-the-art (SOTA) noisy-label-training methods in terms of test accuracy and even outperform sophisticated methods that use clean labels.