# Gradient-based Hyperparameter Optimization Over Long Horizons

arXiv: https://arxiv.org/abs/2007.07869 · HTML: https://mlsanity.com/paper/2007.07869/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jul 2020.

## Built on by (most cited first, 12 of 12)

- [Continuous-Time Meta-Learning with Forward Mode Differentiation](https://mlsanity.com/paper/2203.01443/) (2022)
- [Meta Mirror Descent: Optimiser Learning for Fast Convergence](https://mlsanity.com/paper/2203.02711/) (2022)
- [Optimizing Canaries for Privacy Auditing with Metagradient Descent](https://mlsanity.com/paper/2507.15836/) (2025)
- [MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameters](https://mlsanity.com/paper/2402.02342/) (2024)
- [A Stochastic Approach to Bi-Level Optimization for Hyperparameter Optimization and Meta Learning](https://mlsanity.com/paper/2410.10417/) (2024)
- [Simple and Effective Gradient-Based Tuning of Sequence-to-Sequence Models](https://mlsanity.com/paper/2209.04683/) (2022)
- [Federated Automatic Differentiation](https://mlsanity.com/paper/2301.07806/) (2023)
- [UNEM: UNrolled Generalized EM for Transductive Few-Shot Learning](https://mlsanity.com/paper/2412.16739/) (2024)
- [ULTHO: Ultra-Lightweight yet Efficient Hyperparameter Optimization in Deep Reinforcement Learning](https://mlsanity.com/paper/2503.06101/) (2025)
- [Efficient Hyperparameter Optimization for LLM Reinforcement Learning](https://mlsanity.com/paper/2606.03073/) (2026)
- [Glocal Hypergradient Estimation with Koopman Operator](https://mlsanity.com/paper/2402.02741/) (2024)
- [Reducing Hyperparameter Tuning Costs in ML, Vision and Language Model Training Pipelines via Memoization-Awareness](https://mlsanity.com/paper/2411.03731/) (2024)

## Abstract (arXiv)

Gradient-based hyperparameter optimization has earned a widespread popularity in the context of few-shot meta-learning, but remains broadly impractical for tasks with long horizons (many gradient steps), due to memory scaling and gradient degradation issues. A common workaround is to learn hyperparameters online, but this introduces greediness which comes with a significant performance drop. We propose forward-mode differentiation with sharing (FDS), a simple and efficient algorithm which tackles memory scaling issues with forward-mode differentiation, and gradient degradation issues by sharing hyperparameters that are contiguous in time. We provide theoretical guarantees about the noise reduction properties of our algorithm, and demonstrate its efficiency empirically by differentiating through ~ 104 gradient steps of unrolled optimization. We consider large hyperparameter search ranges on CIFAR-10 where we significantly outperform greedy gradient-based alternatives, while achieving × 20 speedups compared to the state-of-the-art black-box methods. Code is available at: urlhttps://github.com/polo5/FDS