mlsanityData to 7 Oct 2026

Glocal Hypergradient Estimation with Koopman Operator

arXiv 2402.02741 ↗First posted 5 Feb 2024cs.LG stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Gradient-based hyperparameter optimization methods update hyperparameters using hypergradients, gradients of a meta criterion with respect to hyperparameters. Previous research used two distinct update strategies: optimizing hyperparameters using global hypergradients obtained after completing model training or local hypergradients derived after every few model updates. While global hypergradients offer reliability, their computational cost is significant; conversely, local hypergradients provide speed but are often suboptimal. In this paper, we propose *glocal* hypergradient estimation, blending "global" quality with "local" efficiency. To this end, we use the Koopman operator theory to linearize the dynamics of hypergradients so that the global hypergradients can be efficiently approximated only by using a trajectory of local hypergradients. Consequently, we can optimize hyperparameters greedily using estimated global hypergradients, achieving both reliability and efficiency simultaneously. Through numerical experiments of hyperparameter optimization, including optimization of optimizers, we demonstrate the effectiveness of the glocal hypergradient estimation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 35 of its 55 references that are in the map. First posted on arXiv 5 Feb 2024.

Builds on

35 of 55 references.

References · 35

  1. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  4. 2016Wide Residual Networkscited by 3,057
  5. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  6. 2018Group Normalizationcited by 1,514
  7. 2019Class-Balanced Loss Based on Effective Number of Samplescited by 1,239
  8. 2019Meta-Learning with Implicit Gradientscited by 523
  9. 2018Deep Learning for Classical Japanese Literaturecited by 415
  10. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 357
  11. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
  12. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 303
23 more
  1. 2021Modern Koopman Theory for Dynamical Systemscited by 269
  2. 2017Forward and Reverse Gradient-Based Hyperparameter Optimizationcited by 232
  3. 2016Hyperparameter optimization with approximate gradientcited by 231
  4. 2021Efficient and Modular Implicit Differentiationcited by 199
  5. 2021Equinox: neural networks in JAX via callable PyTrees and filtered transformationscited by 143
  6. 2019Generalized Inner Loop Meta-Learningcited by 92
  7. 2015Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameterscited by 71
  8. 2017On Convergence of Extended Dynamic Mode Decomposition to the Koopman Operatorcited by 70
  9. 2018Understanding Short-Horizon Bias in Stochastic Meta-Optimizationcited by 68
  10. 2021iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradientscited by 51
  11. 2022AutoBalance: Optimized Loss Functions for Imbalanced Datacited by 49
  12. 2020Optimizing Neural Networks via Koopman Operator Theorycited by 32
  13. 2020Meta Approach to Data Augmentation Optimizationcited by 22
  14. 2021An Operator Theoretic View on Pruning Deep Neural Networkscited by 18
  15. 2023Making Scalable Meta Learning Practicalcited by 17
  16. 2020Applications of Koopman Mode Analysis to Neural Networkscited by 12
  17. 2020Gradient-based Hyperparameter Optimization Over Long Horizonscited by 12
  18. 2021An Operator Theoretic Approach for Analyzing Sequence Neural Networkscited by 12
  19. 2022Continuous-Time Meta-Learning with Forward Mode Differentiationcited by 12
  20. 2023Koopman-based generalization bound: New aspect for full-rank weightscited by 9
  21. 2023Nystrom Method for Accurate and Scalable Implicit Differentiationcited by 8
  22. 2019Krylov Subspace Method for Nonlinear Dynamical Systems with Random Noisecited by 7
  23. 2022Stable Invariant Models via Koopman Spectracited by 4

Built on by

Not cited yet.