mlsanityData to 14 Oct 2026

Forward and Reverse Gradient-Based Hyperparameter Optimization

arXiv 1703.01785 ↗First posted 6 Mar 2017stat.MLcited by 234

Not on the map. Why?

Abstract (arXiv)

We study two procedures (reverse-mode and forward-mode) for computing the gradient of the validation error with respect to the hyperparameters of any iterative learning algorithm such as stochastic gradient descent. These procedures mirror two methods of computing gradients for recurrent neural networks and have different trade-offs in terms of running time and space requirements. Our formulation of the reverse-mode procedure is linked to previous work by Maclaurin et al. [2015] but does not require reversible dynamics. The forward-mode procedure is suitable for real-time hyperparameter updates, which may significantly speed up hyperparameter optimization on large datasets. We present experiments on data cleaning and on learning task interactions. We also present one large-scale experiment where the use of previous gradient-based methods would be prohibitive.

Key facts. Cited by 234 papers in the map (data through 14 Oct 2026). First posted on arXiv 6 Mar 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 40; 2022: 48; 2023: 51; 2024: 44; 2025: 30; 2026: 21.

234 papers.

Sort:

By area:

  1. 2021Graph Contrastive Learning Automatedcited by 287
  2. 2021Efficient and Modular Implicit Differentiationcited by 203
  3. 2021Discrete Graph Structure Learning for Forecasting Multiple Time Seriescited by 171
  4. 2021Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyondcited by 153
  5. 2021Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challengescited by 145
  6. 2023Dataset Distillation: A Comprehensive Reviewcited by 127
  7. 2021A Near-Optimal Algorithm for Stochastic Bilevel Optimization via Double-Momentumcited by 106
  8. 2021Provably Faster Algorithms for Bilevel Optimizationcited by 106
  9. 2022BOME! Bilevel Optimization Made Easy: A Simple First-Order Approachcited by 105
  10. 2022Remember the Past: Distilling Datasets into Addressable Memories for Neural Networkscited by 104
  11. 2022A framework for bilevel optimization that enables stochastic and global variance reduction algorithmscited by 99
  12. 2023On Penalty-based Bilevel Gradient Descent Methodcited by 87
38 more
  1. 2022Tutorial on amortized optimizationcited by 83
  2. 2021On Training Implicit Modelscited by 75
  3. 2023Using Large Language Models for Hyperparameter Optimizationcited by 71
  4. 2024SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selectioncited by 64
  5. 2021Towards Gradient-based Bilevel Optimization with Non-convex Followers and Beyondcited by 63
  6. 2021Amortized Implicit Differentiation for Stochastic Bilevel Optimizationcited by 63
  7. 2021A Value-Function-based Interior-point Method for Non-convex Bi-level Optimizationcited by 60
  8. 2022FedNest: Federated Bilevel, Minimax, and Compositional Optimizationcited by 60
  9. 2021An End-to-End Framework for Molecular Conformation Generation via Bilevel Programmingcited by 59
  10. 2023Revisiting Weighted Aggregation in Federated Learning with Neural Networkscited by 58
  11. 2022Advancing Model Pruning via Bi-level Optimizationcited by 56
  12. 2021A Fully Single Loop Algorithm for Bilevel Optimization without Hessian Inversecited by 53
  13. 2021Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategiescited by 52
  14. 2021iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradientscited by 51
  15. 2022Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learningcited by 50
  16. 2021Lower Bounds and Accelerated Algorithms for Bilevel Optimizationcited by 46
  17. 2023Optimal Linear Decay Learning Rate Schedules and Further Refinementscited by 43
  18. 2023First-order penalty methods for bilevel optimizationcited by 41
  19. 2023An Introduction to Bi-level Optimization: Foundations and Applications in Signal Processing and Machine Learningcited by 41
  20. 2024Principled Penalty-based Methods for Bilevel Reinforcement Learning and RLHFcited by 41
  21. 2023On Finding Small Hyper-Gradients in Bilevel Optimization: Hardness Results and Improved Analysiscited by 39
  22. 2021HYDRA: Hypergradient Data Relevance Analysis for Interpreting Deep Neural Networkscited by 38
  23. 2024ScaleBiO: Scalable Bilevel Optimization for LLM Data Reweightingcited by 37
  24. 2022Will Bilevel Optimizers Benefit from Loopscited by 36
  25. 2022Online Knowledge Distillation via Mutual Contrastive Learning for Visual Recognitioncited by 36
  26. 2021On the Convergence Theory for Hessian-Free Bilevel Algorithmscited by 35
  27. 2022Learning the Effect of Registration Hyperparameters with HyperMorphcited by 35
  28. 2023AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Riskscited by 35
  29. 2022Dataset Condensation with Latent Space Knowledge Factorization and Sharingcited by 34
  30. 2021Stability and Generalization of Bilevel Programming in Hyperparameter Optimizationcited by 31
  31. 2023Optimal Algorithms for Stochastic Bilevel Optimization under Relaxed Smoothness Conditionscited by 31
  32. 2021A General Descent Aggregation Framework for Gradient-based Bi-level Optimizationcited by 28
  33. 2022Betty: An Automatic Differentiation Library for Multilevel Optimizationcited by 27
  34. 2023Probabilistic Bilevel Coreset Selectioncited by 27
  35. 2022Gradient-based Bi-level Optimization for Deep Learning: A Surveycited by 26
  36. 2023Averaged Method of Multipliers for Bi-Level Optimization without Lower-Level Strong Convexitycited by 26
  37. 2021Value-Function-based Sequential Minimization for Bi-level Optimizationcited by 24
  38. 2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23

50 most cited of 234 All 234 (JSON)