mlsanityData to 7 Oct 2026

Gradient-based Hyperparameter Optimization through Reversible Learning

arXiv 1502.03492 ↗First posted 11 Feb 2015stat.ML cs.LGcited by 357

Not on the map. Why?

Abstract (arXiv)

Tuning hyperparameters of learning algorithms is hard because gradients are usually unavailable. We compute exact gradients of cross-validation performance with respect to all hyperparameters by chaining derivatives backwards through the entire training procedure. These gradients allow us to optimize thousands of hyperparameters, including step-size and momentum schedules, weight initialization distributions, richly parameterized regularization schemes, and neural network architectures. We compute hyperparameter gradients by exactly reversing the dynamics of stochastic gradient descent with momentum.

Key facts. Cited by 357 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Feb 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 69; 2022: 87; 2023: 69; 2024: 52; 2025: 43; 2026: 37.

357 papers.

Sort:

By area:

  1. 2022Dataset Distillation by Matching Training Trajectoriescited by 437
  2. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 329
  3. 2021Graph Contrastive Learning Automatedcited by 287
  4. 2022Dataset Condensation via Efficient Synthetic-Data Parameterizationcited by 187
  5. 2022Dataset Distillation using Neural Feature Regressioncited by 177
  6. 2021Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyondcited by 151
  7. 2021Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challengescited by 143
  8. 2022Minimizing the Accumulated Trajectory Error to Improve Dataset Distillationcited by 142
  9. 2023Dataset Distillation: A Comprehensive Reviewcited by 126
  10. 2023A Comprehensive Survey of Dataset Distillationcited by 114
  11. 2021HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPOcited by 111
  12. 2022Efficient Dataset Distillation Using Random Feature Approximationcited by 111
38 more
  1. 2021Provably Faster Algorithms for Bilevel Optimizationcited by 106
  2. 2022A framework for bilevel optimization that enables stochastic and global variance reduction algorithmscited by 99
  3. 2022Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoningcited by 93
  4. 2023On Penalty-based Bilevel Gradient Descent Methodcited by 87
  5. 2023Data Distillation: A Surveycited by 86
  6. 2022Learning to Learn with Generative Models of Neural Network Checkpointscited by 85
  7. 2021HyperMorph: Amortized Hyperparameter Learning for Image Registrationcited by 84
  8. 2022Tutorial on amortized optimizationcited by 83
  9. 2021Gradients are Not All You Needcited by 82
  10. 2021On Training Implicit Modelscited by 73
  11. 2022Auto-Lambda: Disentangling Dynamic Task Relationshipscited by 73
  12. 2021Learning Normal Dynamics in Videos with Meta Prototype Networkcited by 71
  13. 2023Using Large Language Models for Hyperparameter Optimizationcited by 71
  14. 2022Theseus: A Library for Differentiable Nonlinear Optimizationcited by 64
  15. 2021Towards Gradient-based Bilevel Optimization with Non-convex Followers and Beyondcited by 62
  16. 2022Accelerating Dataset Distillation via Model Augmentationcited by 62
  17. 2022FedNest: Federated Bilevel, Minimax, and Compositional Optimizationcited by 60
  18. 2021An End-to-End Framework for Molecular Conformation Generation via Bilevel Programmingcited by 59
  19. 2022VeLO: Training Versatile Learned Optimizers by Scaling Upcited by 57
  20. 2023Revisiting Weighted Aggregation in Federated Learning with Neural Networkscited by 57
  21. 2021A Fully Single Loop Algorithm for Bilevel Optimization without Hessian Inversecited by 53
  22. 2025End-to-End Test-Time Training for Long Contextcited by 53
  23. 2021Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategiescited by 52
  24. 2024Erasing Undesirable Influence in Diffusion Modelscited by 52
  25. 2021Soft-Label Anonymous Gastric X-ray Image Distillationcited by 51
  26. 2021iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradientscited by 51
  27. 2022Model-Based Deep Learning: On the Intersection of Deep Learning and Optimizationcited by 50
  28. 2023Class-Incremental Exemplar Compression for Class-Incremental Learningcited by 50
  29. 2022AutoBalance: Optimized Loss Functions for Imbalanced Datacited by 49
  30. 2022Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learningcited by 49
  31. 2021Lower Bounds and Accelerated Algorithms for Bilevel Optimizationcited by 46
  32. 2023Dataset Distillation with Convexified Implicit Gradientscited by 46
  33. 2021Bootstrapped Meta-Learningcited by 45
  34. 2023Model Agnostic Sample Reweighting for Out-of-Distribution Learningcited by 43
  35. 2024Principled Penalty-based Methods for Bilevel Reinforcement Learning and RLHFcited by 41
  36. 2023First-order penalty methods for bilevel optimizationcited by 40
  37. 2023An Introduction to Bi-level Optimization: Foundations and Applications in Signal Processing and Machine Learningcited by 39
  38. 2021HYDRA: Hypergradient Data Relevance Analysis for Interpreting Deep Neural Networkscited by 38

50 most cited of 357 All 357 (JSON)