mlsanityData to 14 Oct 2026

Optimizing Millions of Hyperparameters by Implicit Differentiation

arXiv 1911.02590 ↗First posted 6 Nov 2019cs.LG stat.MLcited by 304

Not on the map. Why?

Abstract (arXiv)

We propose an algorithm for inexpensive gradient-based hyperparameter optimization that combines the implicit function theorem (IFT) with efficient inverse Hessian approximations. We present results about the relationship between the IFT and differentiating through optimization, motivating our algorithm. We use the proposed approach to train modern network architectures with millions of weights and millions of hyper-parameters. For example, we learn a data-augmentation network - where every weight is a hyperparameter tuned for validation performance - outputting augmented training examples. Jointly tuning weights and hyperparameters with our approach is only a few times more costly in memory and compute than standard training.

Key facts. Cited by 304 papers in the map (data through 14 Oct 2026). First posted on arXiv 6 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 53; 2022: 70; 2023: 53; 2024: 60; 2025: 38; 2026: 30.

304 papers.

Sort:

By area:

  1. 2021Efficient and Modular Implicit Differentiationcited by 203
  2. 2022Dataset Distillation using Neural Feature Regressioncited by 177
  3. 2021Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyondcited by 153
  4. 2021Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challengescited by 145
  5. 2023Dataset Distillation: A Comprehensive Reviewcited by 127
  6. 2022Efficient Dataset Distillation Using Random Feature Approximationcited by 111
  7. 2022BOME! Bilevel Optimization Made Easy: A Simple First-Order Approachcited by 105
  8. 2024EraseAnything: Enabling Concept Erasure in Rectified Flow Transformerscited by 95
  9. 2022Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoningcited by 94
  10. 2023DREAM: Efficient Dataset Distillation by Representative Matchingcited by 94
  11. 2021Proximal Optimal Transport Modeling of Population Dynamicscited by 93
  12. 2023Data Distillation: A Surveycited by 87
38 more
  1. 2021JFB: Jacobian-Free Backpropagation for Implicit Networkscited by 86
  2. 2024Better than classical? The subtle art of benchmarking quantum machine learning modelscited by 85
  3. 2022Tutorial on amortized optimizationcited by 83
  4. 2021Is Attention Better Than Matrix Decomposition?cited by 77
  5. 2021On Training Implicit Modelscited by 75
  6. 2021Label-Imbalanced and Group-Sensitive Classification under Overparameterizationcited by 74
  7. 2023Using Large Language Models for Hyperparameter Optimizationcited by 71
  8. 2022Theseus: A Library for Differentiable Nonlinear Optimizationcited by 65
  9. 2021Towards Gradient-based Bilevel Optimization with Non-convex Followers and Beyondcited by 63
  10. 2021Amortized Implicit Differentiation for Stochastic Bilevel Optimizationcited by 63
  11. 2021A Value-Function-based Interior-point Method for Non-convex Bi-level Optimizationcited by 60
  12. 2022FedNest: Federated Bilevel, Minimax, and Compositional Optimizationcited by 60
  13. 2022Improving Object-centric Learning with Query Optimizationcited by 60
  14. 2021Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategiescited by 52
  15. 2021iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradientscited by 51
  16. 2021Meta-learning PINN loss functionscited by 51
  17. 2022Model-Based Deep Learning: On the Intersection of Deep Learning and Optimizationcited by 50
  18. 2022Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learningcited by 50
  19. 2022AutoBalance: Optimized Loss Functions for Imbalanced Datacited by 49
  20. 2022Exploring Example Influence in Continual Learningcited by 44
  21. 2023Model Agnostic Sample Reweighting for Out-of-Distribution Learningcited by 44
  22. 2022Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximationscited by 42
  23. 2022Relaxing Equivariance Constraints with Non-stationary Continuous Filterscited by 41
  24. 2021Deep Proxy Causal Learning and its Application to Confounded Bandit Policy Evaluationcited by 40
  25. 2021HYDRA: Hypergradient Data Relevance Analysis for Interpreting Deep Neural Networkscited by 38
  26. 2024ScaleBiO: Scalable Bilevel Optimization for LLM Data Reweightingcited by 37
  27. 2021Control-Oriented Model-Based Reinforcement Learning with Implicit Differentiationcited by 35
  28. 2021On the Convergence Theory for Hessian-Free Bilevel Algorithmscited by 35
  29. 2023AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Riskscited by 35
  30. 2021Stability and Generalization of Bilevel Programming in Hyperparameter Optimizationcited by 31
  31. 2022On Implicit Bias in Overparameterized Bilevel Optimizationcited by 30
  32. 2021Meta-Calibration: Learning of Model Calibration Using Differentiable Expected Calibration Errorcited by 29
  33. 2022Learning Model Predictive Controllers with Real-Time Attention for Real-World Navigationcited by 29
  34. 2025Think Smarter not Harder: Adaptive Reasoning with Inference Aware Optimizationcited by 29
  35. 2021A General Descent Aggregation Framework for Gradient-based Bi-level Optimizationcited by 28
  36. 2023One-step differentiation of iterative algorithmscited by 28
  37. 2021Meta-Learning to Improve Pre-Trainingcited by 27
  38. 2022Betty: An Automatic Differentiation Library for Multilevel Optimizationcited by 27

50 most cited of 304 All 304 (JSON)