AdaFlood: Adaptive Flood Regularization
Not on the map. Why?
Abstract (arXiv)
Although neural networks are conventionally optimized towards zero training loss, it has been recently learned that targeting a non-zero training loss threshold, referred to as a flood level, often enables better test time generalization. Current approaches, however, apply the same constant flood level to all training samples, which inherently assumes all the samples have the same difficulty. We present AdaFlood, a novel flood regularization method that adapts the flood level of each training sample according to the difficulty of the sample. Intuitively, since training samples are not equal in difficulty, the target training loss should be conditioned on the instance. Experiments on datasets covering four diverse input modalities - text, images, asynchronous event sequences, and tabular - demonstrate the versatility of AdaFlood across data domains and noise levels.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 42 of its 58 references that are in the map. First posted on arXiv 6 Nov 2023.
Builds on
42 of 58 references.
References · 42
- 2017Attention Is All You Needcited by 60,361
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
- 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
- 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
- 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
- 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
- 2018DARTS: Differentiable Architecture Searchcited by 1,798
- 2019Contrastive Multiview Codingcited by 1,301
- 2019Learning Imbalanced Datasets with Label-Distribution-Aware Margin Losscited by 989
30 more
- 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
- 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
- 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
- 2014OpenML: networked science in machine learningcited by 526
- 2018Bilevel Programming for Hyperparameter Optimization and Meta-Learningcited by 405
- 2018Meta-learning with differentiable closed-form solverscited by 397
- 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 381
- 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 342
- 2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 339
- 2020Are we done with ImageNet?cited by 333
- 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 236
- 2020Transformer Hawkes Processcited by 235
- 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
- 2019Intensity-Free Learning of Temporal Point Processescited by 140
- 2013Dropout Training as Adaptive Regularizationcited by 93
- 2020Characterizing Structural Regularities of Labeled Data in Overparameterized Modelscited by 93
- 2021Rethinking Architecture Selection in Differentiable NAScited by 92
- 2019Dynamic Convolutions: Exploiting Spatial Sparsity for Faster Inferencecited by 91
- 2021Dynamic Slimmable Networkcited by 83
- 2020Group Sparsity: The Hinge Between Filter Pruning and Decomposition for Network Compressioncited by 72
- 2022The Effects of Regularization and Data Augmentation are Class Dependentcited by 67
- 2019Regularizing Activation Distribution for Training Binarized Deep Networkscited by 61
- 2020Do We Need Zero Training Loss After Achieving Zero Training Error?cited by 61
- 2023Towards Last-layer Retraining for Group Robustness with Fewer Annotationscited by 54
- 2022RankSim: Ranking Similarity Regularization for Deep Imbalanced Regressioncited by 52
- 2022Characterizing Datapoints via Second-Split Forgettingcited by 37
- 2022A Fast, Well-Founded Approximation to the Empirical Neural Tangent Kernelcited by 30
- 2021Noisy Feature Mixupcited by 27
- 2022Better Supervisory Signals by Observing Learning Pathscited by 21
- 2023Meta Temporal Point Processescited by 17
Built on by
Citing papers per year: 2025: 2.
2 papers.