mlsanityData to 13 Oct 2026

Training Deep Neural Networks via Branch-and-Bound

arXiv 2104.01730 ↗First posted 5 Apr 2021cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose BPGrad, a novel approximate algorithm for deep nueral network training, based on adaptive estimates of feasible region via branch-and-bound. The method is based on the assumption of Lipschitz continuity in objective function, and as a result, it can adaptively determine the step size for the current gradient given the history of previous updates. We prove that, by repeating such a branch-and-pruning procedure, it can achieve the optimal solution within finite iterations. A computationally efficient solver based on BPGrad has been proposed to train the deep neural networks. Empirical results demonstrate that BPGrad solver works well in practice and compares favorably to other stochastic optimization methods in the tasks of object recognition, detection, and segmentation. The code is available at urlhttps://github.com/RyanCV/BPGrad.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 51 of its 73 references that are in the map. First posted on arXiv 5 Apr 2021.

Builds on

51 of 73 references.

References · 51

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,321
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,625
  6. 2017Mask R-CNNcited by 7,266
  7. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,849
  8. 2016Wide Residual Networkscited by 3,063
  9. 2015Fast R-CNNcited by 2,933
  10. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,162
  11. 2016Understanding deep learning requires rethinking generalizationcited by 1,666
  12. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,344
38 more
  1. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,142
  2. 2017Visualizing the Loss Landscape of Neural Netscited by 1,129
  3. 2016An overview of gradient descent optimization algorithmscited by 1,028
  4. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 786
  5. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 658
  6. 2017Sharp Minima Can Generalize For Deep Netscited by 465
  7. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 358
  8. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  9. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 329
  10. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 288
  11. 2014The Loss Surfaces of Multilayer Networkscited by 280
  12. 2016Deep Learning without Poor Local Minimacited by 234
  13. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 140
  14. 2017Convergence Analysis of Two-layer Neural Networks with ReLU Activationcited by 137
  15. 2016Topology and Geometry of Half-Rectified Network Optimizationcited by 102
  16. 2014Deep learning with Elastic Averaging SGDcited by 90
  17. 2018ResNet with one-neuron hidden layers is a Universal Approximatorcited by 78
  18. 2017The loss surface of deep and wide neural networkscited by 74
  19. 2017Globally Optimal Gradient Descent for a ConvNet with Gaussian Inputscited by 61
  20. 2017Variants of RMSProp and Adagrad with Logarithmic Regret Boundscited by 50
  21. 2019Recent Advances in Deep Learning for Object Detectioncited by 50
  22. 2019Training Neural Networks for and by Interpolationcited by 45
  23. 2017Global optimization of Lipschitz functionscited by 39
  24. 2019SGD Converges to Global Minimum in Deep Learning via Star-convex Pathcited by 39
  25. 2017Global Guarantees for Enforcing Deep Generative Priors by Empirical Riskcited by 37
  26. 2014MatConvNet - Convolutional Neural Networks for MATLABcited by 36
  27. 2016Gradient Descent Only Converges to Minimizers: Non-Isolated Critical Points and Invariant Regionscited by 34
  28. 2021SOSD-Net: Joint Semantic Object Segmentation and Depth Estimation from Monocular imagescited by 33
  29. 2017Global optimality conditions for deep neural networkscited by 31
  30. 2016No bad local minima: Data independent training error guarantees for multilayer neural networkscited by 29
  31. 2017Convergent Block Coordinate Descent for Training Tikhonov Regularized Deep Neural Networkscited by 26
  32. 2018Adding One Neuron Can Eliminate All Bad Local Minimacited by 21
  33. 2020Deep Feature Augmentation for Occluded Image Classificationcited by 21
  34. 2019MDFN: Multi-Scale Deep Feature Learning Network for Object Detectioncited by 19
  35. 2018Deep Frank-Wolfe For Neural Network Optimizationcited by 11
  36. 2017BPGrad: Towards Global Optimality in Deep Learning via Branch and Pruningcited by 5
  37. 2019Unsupervised Deep Feature Transfer for Low Resolution Image Classificationcited by 5
  38. 2020Self-Orthogonality Module: A Network Architecture Plug-in for Learning Orthogonal Filterscited by 5

50 most cited of 51 All 51 (JSON)

Built on by

Not cited yet.