mlsanityData to 7 Oct 2026

Cogradient Descent for Dependable Learning

arXiv 2106.10617 ↗First posted 20 Jun 2021cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Conventional gradient descent methods compute the gradients for multiple variables through the partial derivative. Treating the coupled variables independently while ignoring the interaction, however, leads to an insufficient optimization for bilinear models. In this paper, we propose a dependable learning based on Cogradient Descent (CoGD) algorithm to address the bilinear optimization problem, providing a systematic way to coordinate the gradients of coupling variables based on a kernelized projection function. CoGD is introduced to solve bilinear problems when one variable is with sparsity constraint, as often occurs in modern learning paradigms. CoGD can also be used to decompose the association of features and weights, which further generalizes our method to better train convolutional neural networks (CNNs) and improve the model capacity. CoGD is applied in representative bilinear problems, including image reconstruction, image inpainting, network pruning and CNN training. Extensive experiments show that CoGD improves the state-of-the-arts by significant margins. Code is available at {https://github.com/bczhangbczhang/CoGD}.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 38 of its 65 references that are in the map. First posted on arXiv 20 Jun 2021.

Builds on

38 of 65 references.

References · 38

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  6. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  7. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  8. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  9. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  10. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  11. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  12. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 703
26 more
  1. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  2. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
  3. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 438
  4. 2018Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Accelerationcited by 365
  5. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 304
  6. 2020HRank: Filter Pruning using High-Rank Feature Mapcited by 291
  7. 2016Dynamic Network Surgery for Efficient DNNscited by 233
  8. 2015Bilinear CNNs for Fine-grained Visual Recognitioncited by 228
  9. 2017NISP: Pruning Networks using Neuron Importance Score Propagationcited by 228
  10. 2016Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architecturescited by 220
  11. 2015Quantized Convolutional Neural Networks for Mobile Devicescited by 212
  12. 2017Data-Driven Sparse Structure Selection for Deep Neural Networkscited by 181
  13. 2019Towards Optimal Structured CNN Pruning via Generative Adversarial Learningcited by 145
  14. 2016Designing Energy-Efficient Convolutional Neural Networks using Energy-Aware Pruningcited by 133
  15. 2017CondenseNet: An Efficient DenseNet using Learned Group Convolutionscited by 121
  16. 2017Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answeringcited by 117
  17. 2018Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layerscited by 112
  18. 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
  19. 2019Noise Flow: Noise Modeling with Conditional Normalizing Flowscited by 108
  20. 2018Part-Aligned Bilinear Representations for Person Re-identificationcited by 90
  21. 2019Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated Structurecited by 76
  22. 2018Exploiting Kernel Sparsity and Entropy for Interpretable CNN Compressioncited by 40
  23. 2018Accelerate CNN via Recursive Bayesian Pruningcited by 27
  24. 2018Structured Pruning of Neural Networks with Budget-Aware Regularizationcited by 26
  25. 2016Factorized Bilinear Models for Image Recognitioncited by 13
  26. 2020Cogradient Descent for Bilinear Optimizationcited by 3

Built on by

Not cited yet.