# Reducing Bias in Deep Learning Optimization: The RSGDM Approach

arXiv: https://arxiv.org/abs/2409.15314 · HTML: https://mlsanity.com/paper/2409.15314/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 27 references that are in the map.
- First posted on arXiv 5 Sep 2024.

## Builds on (most cited first, 16 of 16)

- [On the Convergence of Adam and Beyond](https://mlsanity.com/paper/1904.09237/) (2019)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [NDC-Scene: Boost Monocular 3D Semantic Scene Completion in Normalized Device Coordinates Space](https://mlsanity.com/paper/2309.14616/) (2023)
- [Survival Prediction Across Diverse Cancer Types Using Neural Networks](https://mlsanity.com/paper/2404.08713/) (2024)
- [Convolutional neural network classification of cancer cytopathology images: taking breast cancer as an example](https://mlsanity.com/paper/2404.08279/) (2024)
- [Attention Mechanism and Context Modeling System for Text Mining Machine Translation](https://mlsanity.com/paper/2408.04216/) (2024)
- [Efficiency optimization of large-scale language models based on deep learning in natural language processing tasks](https://mlsanity.com/paper/2405.11704/) (2024)
- [Application of Multimodal Fusion Deep Learning Model in Disease Recognition](https://mlsanity.com/paper/2406.18546/) (2024)
- [A comparative study of generative adversarial networks for image recognition algorithms based on deep learning and traditional methods](https://mlsanity.com/paper/2408.03568/) (2024)
- [Enhancing Medical Imaging with GANs Synthesizing Realistic Images from Limited Data](https://mlsanity.com/paper/2406.18547/) (2024)
- [An Enhanced Encoder-Decoder Network Architecture for Reducing Information Loss in Image Semantic Segmentation](https://mlsanity.com/paper/2406.01605/) (2024)
- [Exploration of Multi-Scale Image Fusion Systems in Intelligent Medical Image Analysis](https://mlsanity.com/paper/2406.18548/) (2024)
- [Research on Deep Learning Model of Feature Extraction Based on Convolutional Neural Network](https://mlsanity.com/paper/2406.08837/) (2024)
- [Advancements in Feature Extraction Recognition of Medical Imaging Systems Through Deep Learning Technique](https://mlsanity.com/paper/2406.18549/) (2024)
- [Research on Disease Prediction Model Construction Based on Computer AI deep Learning Technology](https://mlsanity.com/paper/2406.16982/) (2024)
- [Research on Feature Extraction Data Processing System For MRI of Brain Diseases Based on Computer Deep Learning](https://mlsanity.com/paper/2406.16981/) (2024)

## Built on by (most cited first, 7 of 7)

- [Metric Learning for Tag Recommendation: Tackling Data Sparsity and Cold Start Issues](https://mlsanity.com/paper/2411.06374/) (2024)
- [Optimizing YOLOv5s Object Detection through Knowledge Distillation algorithm](https://mlsanity.com/paper/2410.12259/) (2024)
- [Adversarial Neural Networks in Medical Imaging Advancements and Challenges in Semantic Segmentation](https://mlsanity.com/paper/2410.13099/) (2024)
- [Balancing Innovation and Privacy: Data Security Strategies in Natural Language Processing Applications](https://mlsanity.com/paper/2410.08553/) (2024)
- [A Recommendation Model Utilizing Separation Embedding and Self-Attention for Feature Mining](https://mlsanity.com/paper/2410.15026/) (2024)
- [Deep Learning for Medical Text Processing: BERT Model Fine-Tuning and Comparative Study](https://mlsanity.com/paper/2410.20792/) (2024)
- [Automated Genre-Aware Article Scoring and Feedback Using Large Language Models](https://mlsanity.com/paper/2410.14165/) (2024)

## Abstract (arXiv)

Currently, widely used first-order deep learning optimizers include non-adaptive learning rate optimizers and adaptive learning rate optimizers. The former is represented by SGDM (Stochastic Gradient Descent with Momentum), while the latter is represented by Adam. Both of these methods use exponential moving averages to estimate the overall gradient. However, estimating the overall gradient using exponential moving averages is biased and has a lag. This paper proposes an RSGDM algorithm based on differential correction. Our contributions are mainly threefold: 1) Analyze the bias and lag brought by the exponential moving average in the SGDM algorithm. 2) Use the differential estimation term to correct the bias and lag in the SGDM algorithm, proposing the RSGDM algorithm. 3) Experiments on the CIFAR datasets have proven that our RSGDM algorithm is superior to the SGDM algorithm in terms of convergence accuracy.