# Enhancing Deep Learning with Optimized Gradient Descent: Bridging Numerical Methods and Neural Network Training

arXiv: https://arxiv.org/abs/2409.04707 · HTML: https://mlsanity.com/paper/2409.04707/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 33 references that are in the map.
- First posted on arXiv 7 Sep 2024.

## Builds on (most cited first, 20 of 20)

- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [SGD: General Analysis and Improved Rates](https://mlsanity.com/paper/1901.09401/) (2019)
- [Do CIFAR-10 Classifiers Generalize to CIFAR-10?](https://mlsanity.com/paper/1806.00451/) (2018)
- [AdaGrad stepsizes: Sharp convergence over nonconvex landscapes](https://mlsanity.com/paper/1806.01811/) (2018)
- [A Sufficient Condition for Convergences of Adam and RMSProp](https://mlsanity.com/paper/1811.09358/) (2018)
- [Improving Generalization Performance by Switching from Adam to SGD](https://mlsanity.com/paper/1712.07628/) (2017)
- [NDC-Scene: Boost Monocular 3D Semantic Scene Completion in Normalized Device Coordinates Space](https://mlsanity.com/paper/2309.14616/) (2023)
- [Survival Prediction Across Diverse Cancer Types Using Neural Networks](https://mlsanity.com/paper/2404.08713/) (2024)
- [Convolutional neural network classification of cancer cytopathology images: taking breast cancer as an example](https://mlsanity.com/paper/2404.08279/) (2024)
- [Exploring Diverse Methods in Visual Question Answering](https://mlsanity.com/paper/2404.13565/) (2024)
- [The FacT: Taming Latent Factor Models for Explainability with Factorization Trees](https://mlsanity.com/paper/1906.02037/) (2019)
- [Enhance Image-to-Image Generation with LLaVA-generated Prompts](https://mlsanity.com/paper/2406.01956/) (2024)
- [Recurrent Neural Network from Adder's Perspective: Carry-lookahead RNN](https://mlsanity.com/paper/2106.12901/) (2021)
- [Attention Mechanism and Context Modeling System for Text Mining Machine Translation](https://mlsanity.com/paper/2408.04216/) (2024)
- [A comparative study of generative adversarial networks for image recognition algorithms based on deep learning and traditional methods](https://mlsanity.com/paper/2408.03568/) (2024)
- [Dynamic Hypergraph-Enhanced Prediction of Sequential Medical Visits](https://mlsanity.com/paper/2408.07084/) (2024)
- [Infrared Image Super-Resolution via Lightweight Information Split Network](https://mlsanity.com/paper/2405.10561/) (2024)
- [Deep CNN Ensemble with Data Augmentation for Object Detection](https://mlsanity.com/paper/1506.07224/) (2015)
- [Neural Attentive Bag-of-Entities Model for Text Classification](https://mlsanity.com/paper/1909.01259/) (2019)

## Built on by (most cited first, 7 of 7)

- [Self-Supervised Graph Neural Networks for Enhanced Feature Extraction in Heterogeneous Information Networks](https://mlsanity.com/paper/2410.17617/) (2024)
- [Wasserstein Distance-Weighted Adversarial Network for Cross-Domain Credit Risk Assessment](https://mlsanity.com/paper/2409.18544/) (2024)
- [Contrastive Learning for Knowledge-Based Question Generation in Large Language Models](https://mlsanity.com/paper/2409.13994/) (2024)
- [Optimizing News Text Classification with Bi-LSTM and Attention Mechanism for Efficient Data Processing](https://mlsanity.com/paper/2409.15576/) (2024)
- [Automated Genre-Aware Article Scoring and Feedback Using Large Language Models](https://mlsanity.com/paper/2410.14165/) (2024)
- [Deep Learning-Based Channel Squeeze U-Structure for Lung Nodule Detection and Segmentation](https://mlsanity.com/paper/2409.13868/) (2024)
- [Graph Neural Network Framework for Sentiment Analysis Using Syntactic Feature](https://mlsanity.com/paper/2409.14000/) (2024)

## Abstract (arXiv)

Optimization theory serves as a pivotal scientific instrument for achieving optimal system performance, with its origins in economic applications to identify the best investment strategies for maximizing benefits. Over the centuries, from the geometric inquiries of ancient Greece to the calculus contributions by Newton and Leibniz, optimization theory has significantly advanced. The persistent work of scientists like Lagrange, Cauchy, and von Neumann has fortified its progress. The modern era has seen an unprecedented expansion of optimization theory applications, particularly with the growth of computer science, enabling more sophisticated computational practices and widespread utilization across engineering, decision analysis, and operations research. This paper delves into the profound relationship between optimization theory and deep learning, highlighting the omnipresence of optimization problems in the latter. We explore the gradient descent algorithm and its variants, which are the cornerstone of optimizing neural networks. The chapter introduces an enhancement to the SGD optimizer, drawing inspiration from numerical optimization methods, aiming to enhance interpretability and accuracy. Our experiments on diverse deep learning tasks substantiate the improved algorithm's efficacy. The paper concludes by emphasizing the continuous development of optimization theory and its expanding role in solving intricate problems, enhancing computational capabilities, and informing better policy decisions.