mlsanityData to 14 Oct 2026

Stochastic Re-weighted Gradient Descent via Distributionally Robust Optimization

arXiv 2306.09222 ↗First posted 15 Jun 2023cs.LG cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

We present Re-weighted Gradient Descent (RGD), a novel optimization technique that improves the performance of deep neural networks through dynamic sample re-weighting. Leveraging insights from distributionally robust optimization (DRO) with Kullback-Leibler divergence, our method dynamically assigns importance weights to training data during each optimization step. RGD is simple to implement, computationally efficient, and compatible with widely used optimizers such as SGD and Adam. We demonstrate the effectiveness of RGD on various learning tasks, including supervised learning, meta-learning, and out-of-domain generalization. Notably, RGD achieves state-of-the-art results on diverse benchmarks, with improvements of +0.7% on DomainBed, +1.44% on tabular classification, textcolorblue+1.94% on GLUE with BERT, and +1.01% on ImageNet-1K with ViT.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 73 of its 136 references that are in the map. First posted on arXiv 15 Jun 2023.

Builds on

73 of 136 references.

References · 73

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  4. 2015Domain-Adversarial Training of Neural Networkscited by 3,044
  5. 2017Prototypical Networks for Few-shot Learningcited by 3,015
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  7. 2019Invariant Risk Minimizationcited by 1,854
  8. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  9. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,247
  10. 2019Class-Balanced Loss Based on Effective Number of Samplescited by 1,245
  11. 2016Deep CORAL: Correlation Alignment for Deep Domain Adaptationcited by 1,200
  12. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,059
38 more
  1. 2020In Search of Lost Domain Generalizationcited by 1,054
  2. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  3. 2016An overview of gradient descent optimization algorithmscited by 1,033
  4. 2017Learning to Generalize: Meta-Learning for Domain Generalizationcited by 763
  5. 2020Out-of-Distribution Generalization via Risk Extrapolation (REx)cited by 742
  6. 2018Learning to Reweight Examples for Robust Deep Learningcited by 675
  7. 2017MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labelscited by 653
  8. 2019TabNet: Attentive Interpretable Tabular Learningcited by 648
  9. 2016Training Region-based Object Detectors with Online Hard Example Miningcited by 579
  10. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 434
  11. 2020Self-Challenging Improves Cross-Domain Generalizationcited by 426
  12. 2017Certifying Some Distributional Robustness with Principled Adversarial Trainingcited by 337
  13. 2018Learning Models with Uniform Performance via Distributionally Robust Optimizationcited by 335
  14. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 328
  15. 2014Training Deep Neural Networks on Noisy Labels with Bootstrappingcited by 322
  16. 2021Curriculum Learning: A Surveycited by 259
  17. 2018Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noisecited by 216
  18. 2020Improve Unsupervised Domain Adaptation with Mixup Trainingcited by 185
  19. 2016Statistics of Robust Optimization: A Generalized Empirical Likelihood Approachcited by 183
  20. 2020Adaptive Risk Minimization: Learning to Adapt to Domain Shiftcited by 183
  21. 2017Domain Generalization by Marginal Transfer Learningcited by 180
  22. 2018Dimensionality-Driven Learning with Noisy Labelscited by 173
  23. 2016Variance-based regularization with convex objectivescited by 155
  24. 2022Domain Generalization by Mutual-Information Regularization with Pre-trained Modelscited by 133
  25. 2019Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processescited by 131
  26. 2014Learning to Executecited by 106
  27. 2020Tilted Empirical Risk Minimizationcited by 106
  28. 2021SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learningcited by 104
  29. 2018Tight Analyses for Non-Smooth Stochastic Gradient Descentcited by 97
  30. 2015Webly Supervised Learning of Convolutional Networkscited by 95
  31. 2023Revisiting Gradient Clipping: Stochastic bias and tight convergence guaranteescited by 83
  32. 2020Towards Domain-Agnostic Contrastive Learningcited by 75
  33. 2021Learning Fast Sample Re-weighting Without Reward Datacited by 69
  34. 2017Class Rectification Hard Mining for Imbalanced Deep Learningcited by 67
  35. 2014Curriculum Learning of Multiple Taskscited by 66
  36. 2022PolyLoss: A Polynomial Expansion Perspective of Classification Loss Functionscited by 63
  37. 2021DORO: Distributional and Outlier Robust Optimizationcited by 56
  38. 2017How hard can it be? Estimating the difficulty of visual search in an imagecited by 52

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 1; 2025: 2.

6 papers.

  1. 2025Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretrainingcited by 16
  2. 2023Fairness under Covariate Shift: Improving Fairness-Accuracy tradeoff with few Unlabeled Test Samplescited by 2
  3. 2024The effect of Leaky ReLUs on the training and generalization of overparameterized networkscited by 2
  4. 2023EHI: End-to-end Learning of Hierarchical Index for Efficient Dense Retrievalcited by 1
  5. 2023Responsible AI (RAI) Games and Ensemblescited by 1
  6. 2025Data-Efficient Training by Evolved Samplingcited by 0