mlsanityData to 14 Oct 2026

Controlled Descent Training

arXiv 2303.09216 ↗First posted 16 Mar 2023math.OC cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

In this work, a novel and model-based artificial neural network (ANN) training method is developed supported by optimal control theory. The method augments training labels in order to robustly guarantee training loss convergence and improve training convergence rate. Dynamic label augmentation is proposed within the framework of gradient descent training where the convergence of training loss is controlled. First, we capture the training behavior with the help of empirical Neural Tangent Kernels (NTK) and borrow tools from systems and control theory to analyze both the local and global training dynamics (e.g. stability, reachability). Second, we propose to dynamically alter the gradient descent training mechanism via fictitious labels as control inputs and an optimal state feedback policy. In this way, we enforce locally H2 optimal and convergent training behavior. The novel algorithm, textitControlled Descent Training (CDT), guarantees local convergence. CDT unleashes new potentials in the analysis, interpretation, and design of ANN architectures. The applicability of the method is demonstrated on standard regression and classification problems.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 20 of its 45 references that are in the map. First posted on arXiv 16 Mar 2023.

Builds on

20 of 45 references.

References · 20

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  3. 2020When and why PINNs fail to train: A neural tangent kernel perspectivecited by 634
  4. 2020Infinite attention: NNGP and NTK for deep attention networkscited by 121
  5. 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
  6. 2020Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networkscited by 87
  7. 2020The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Trainingcited by 84
  8. 2020The Recurrent Neural Tangent Kernelcited by 67
  9. 2022Fast Finite Width Neural Tangent Kernelcited by 62
  10. 2021Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamicscited by 57
  11. 2020On the infinite width limit of neural networks with a standard parameterizationcited by 38
  12. 2021Scaling Neural Tangent Kernels via Sketching and Random Featurescited by 32
8 more
  1. 2021A Neural Tangent Kernel Perspective of GANscited by 21
  2. 2019The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Studycited by 16
  3. 2021Short-term traffic prediction using physics-aware neural networkscited by 8
  4. 2020Collegial Ensemblescited by 6
  5. 2022On the Interpretability of Regularisation for Neural Networks Through Model Gradient Similaritycited by 4
  6. 2021Parameter and density estimation from real-world traffic data: A kinetic compartmental approachcited by 3
  7. 2021Constrained Policy Gradient Method for Safe and Fast Reinforcement Learning: a Neural Tangent Kernel Based Approachcited by 2
  8. 2022Deep Q-learning: a robust control approachcited by 1

Built on by

Not cited yet.