mlsanityData to 7 Oct 2026

Reducing Bias in Deep Learning Optimization: The RSGDM Approach

arXiv 2409.15314 ↗First posted 5 Sep 2024cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Currently, widely used first-order deep learning optimizers include non-adaptive learning rate optimizers and adaptive learning rate optimizers. The former is represented by SGDM (Stochastic Gradient Descent with Momentum), while the latter is represented by Adam. Both of these methods use exponential moving averages to estimate the overall gradient. However, estimating the overall gradient using exponential moving averages is biased and has a lag. This paper proposes an RSGDM algorithm based on differential correction. Our contributions are mainly threefold: 1) Analyze the bias and lag brought by the exponential moving average in the SGDM algorithm. 2) Use the differential estimation term to correct the bias and lag in the SGDM algorithm, proposing the RSGDM algorithm. 3) Experiments on the CIFAR datasets have proven that our RSGDM algorithm is superior to the SGDM algorithm in terms of convergence accuracy.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 16 of its 27 references that are in the map. First posted on arXiv 5 Sep 2024.

Builds on

16 of 27 references.

References · 16

  1. 2019On the Convergence of Adam and Beyondcited by 834
  2. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  3. 2023NDC-Scene: Boost Monocular 3D Semantic Scene Completion in Normalized Device Coordinates Spacecited by 88
  4. 2024Survival Prediction Across Diverse Cancer Types Using Neural Networkscited by 80
  5. 2024Convolutional neural network classification of cancer cytopathology images: taking breast cancer as an examplecited by 50
  6. 2024Attention Mechanism and Context Modeling System for Text Mining Machine Translationcited by 15
  7. 2024Efficiency optimization of large-scale language models based on deep learning in natural language processing taskscited by 14
  8. 2024Application of Multimodal Fusion Deep Learning Model in Disease Recognitioncited by 12
  9. 2024A comparative study of generative adversarial networks for image recognition algorithms based on deep learning and traditional methodscited by 12
  10. 2024Enhancing Medical Imaging with GANs Synthesizing Realistic Images from Limited Datacited by 11
  11. 2024An Enhanced Encoder-Decoder Network Architecture for Reducing Information Loss in Image Semantic Segmentationcited by 10
  12. 2024Exploration of Multi-Scale Image Fusion Systems in Intelligent Medical Image Analysiscited by 10
4 more
  1. 2024Research on Deep Learning Model of Feature Extraction Based on Convolutional Neural Networkcited by 7
  2. 2024Advancements in Feature Extraction Recognition of Medical Imaging Systems Through Deep Learning Techniquecited by 7
  3. 2024Research on Disease Prediction Model Construction Based on Computer AI deep Learning Technologycited by 6
  4. 2024Research on Feature Extraction Data Processing System For MRI of Brain Diseases Based on Computer Deep Learningcited by 5

Built on by

Cited by year

Citing papers per year: 2024: 7.

7 papers.

  1. 2024Metric Learning for Tag Recommendation: Tackling Data Sparsity and Cold Start Issuescited by 12
  2. 2024Optimizing YOLOv5s Object Detection through Knowledge Distillation algorithmcited by 11
  3. 2024Adversarial Neural Networks in Medical Imaging Advancements and Challenges in Semantic Segmentationcited by 10
  4. 2024Balancing Innovation and Privacy: Data Security Strategies in Natural Language Processing Applicationscited by 8
  5. 2024A Recommendation Model Utilizing Separation Embedding and Self-Attention for Feature Miningcited by 8
  6. 2024Deep Learning for Medical Text Processing: BERT Model Fine-Tuning and Comparative Studycited by 8
  7. 2024Automated Genre-Aware Article Scoring and Feedback Using Large Language Modelscited by 7