# Adaptive Friction in Deep Learning: Enhancing Optimizers with Sigmoid and Tanh Function

arXiv: https://arxiv.org/abs/2408.11839 · HTML: https://mlsanity.com/paper/2408.11839/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 25 references that are in the map.
- First posted on arXiv 7 Aug 2024.

## Builds on (most cited first, 11 of 11)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [AdaViT: Adaptive Tokens for Efficient Vision Transformer](https://mlsanity.com/paper/2112.07658/) (2021)
- [Do CIFAR-10 Classifiers Generalize to CIFAR-10?](https://mlsanity.com/paper/1806.00451/) (2018)
- [Survival Prediction Across Diverse Cancer Types Using Neural Networks](https://mlsanity.com/paper/2404.08713/) (2024)
- [Exploration of Attention Mechanism-Enhanced Deep Learning Models in the Mining of Medical Textual Data](https://mlsanity.com/paper/2406.00016/) (2024)
- [AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks](https://mlsanity.com/paper/2105.10190/) (2021)
- [Research on Deep Learning Model of Feature Extraction Based on Convolutional Neural Network](https://mlsanity.com/paper/2406.08837/) (2024)
- [Research on Early Warning Model of Cardiovascular Disease Based on Computer Deep Learning](https://mlsanity.com/paper/2406.08864/) (2024)
- [Research on Disease Prediction Model Construction Based on Computer AI deep Learning Technology](https://mlsanity.com/paper/2406.16982/) (2024)

## Built on by (most cited first, 16 of 16)

- [Self-Supervised Graph Neural Networks for Enhanced Feature Extraction in Heterogeneous Information Networks](https://mlsanity.com/paper/2410.17617/) (2024)
- [Adversarial Neural Networks in Medical Imaging Advancements and Challenges in Semantic Segmentation](https://mlsanity.com/paper/2410.13099/) (2024)
- [Unveiling the Potential of Graph Neural Networks in SME Credit Risk Assessment](https://mlsanity.com/paper/2409.17909/) (2024)
- [Contrastive Learning for Knowledge-Based Question Generation in Large Language Models](https://mlsanity.com/paper/2409.13994/) (2024)
- [Applying Hybrid Graph Neural Networks to Strengthen Credit Risk Analysis](https://mlsanity.com/paper/2410.04283/) (2024)
- [Balancing Innovation and Privacy: Data Security Strategies in Natural Language Processing Applications](https://mlsanity.com/paper/2410.08553/) (2024)
- [A Recommendation Model Utilizing Separation Embedding and Self-Attention for Feature Mining](https://mlsanity.com/paper/2410.15026/) (2024)
- [Deep Learning for Medical Text Processing: BERT Model Fine-Tuning and Comparative Study](https://mlsanity.com/paper/2410.20792/) (2024)
- [Optimizing News Text Classification with Bi-LSTM and Attention Mechanism for Efficient Data Processing](https://mlsanity.com/paper/2409.15576/) (2024)
- [Automated Genre-Aware Article Scoring and Feedback Using Large Language Models](https://mlsanity.com/paper/2410.14165/) (2024)
- [Deep Learning with HM-VGG: AI Strategies for Multi-modal Image Analysis](https://mlsanity.com/paper/2410.24046/) (2024)
- [Deep Learning-Based Channel Squeeze U-Structure for Lung Nodule Detection and Segmentation](https://mlsanity.com/paper/2409.13868/) (2024)
- [Graph Neural Network Framework for Sentiment Analysis Using Syntactic Feature](https://mlsanity.com/paper/2409.14000/) (2024)
- [Axial Attention Transformer Networks: A New Frontier in Breast Cancer Detection](https://mlsanity.com/paper/2409.12347/) (2024)
- [Few-Shot Learning with Adaptive Weight Masking in Conditional GANs](https://mlsanity.com/paper/2412.03105/) (2024)
- [Analysis of Financial Risk Behavior Prediction Using Deep Learning and Big Data Algorithms](https://mlsanity.com/paper/2410.19394/) (2024)

## Abstract (arXiv)

Adaptive optimizers are pivotal in guiding the weight updates of deep neural networks, yet they often face challenges such as poor generalization and oscillation issues. To counter these, we introduce sigSignGrad and tanhSignGrad, two novel optimizers that integrate adaptive friction coefficients based on the Sigmoid and Tanh functions, respectively. These algorithms leverage short-term gradient information, a feature overlooked in traditional Adam variants like diffGrad and AngularGrad, to enhance parameter updates and convergence.Our theoretical analysis demonstrates the wide-ranging adjustment capability of the friction coefficient S, which aligns with targeted parameter update strategies and outperforms existing methods in both optimization trajectory smoothness and convergence rate. Extensive experiments on CIFAR-10, CIFAR-100, and Mini-ImageNet datasets using ResNet50 and ViT architectures confirm the superior performance of our proposed optimizers, showcasing improved accuracy and reduced training time. The innovative approach of integrating adaptive friction coefficients as plug-ins into existing optimizers, exemplified by the sigSignAdamW and sigSignAdamP variants, presents a promising strategy for boosting the optimization performance of established algorithms. The findings of this study contribute to the advancement of optimizer design in deep learning.