# DrMAD: Distilling Reverse-Mode Automatic Differentiation for Optimizing Hyperparameters of Deep Neural Networks

arXiv: https://arxiv.org/abs/1601.00917 · HTML: https://mlsanity.com/paper/1601.00917/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jan 2016.

## Built on by (most cited first, 9 of 9)

- [Learning the Effect of Registration Hyperparameters with HyperMorph](https://mlsanity.com/paper/2203.16680/) (2022)
- [Stability and Generalization of Bilevel Programming in Hyperparameter Optimization](https://mlsanity.com/paper/2106.04188/) (2021)
- [Flexora: Flexible Low Rank Adaptation for Large Language Models](https://mlsanity.com/paper/2408.10774/) (2024)
- [Adaptive Gradient Methods with Local Guarantees](https://mlsanity.com/paper/2203.01400/) (2022)
- [Differentiable Model Selection for Ensemble Learning](https://mlsanity.com/paper/2211.00251/) (2022)
- [Online Hyperparameter Meta-Learning with Hypergradient Distillation](https://mlsanity.com/paper/2110.02508/) (2021)
- [Simple and Effective Gradient-Based Tuning of Sequence-to-Sequence Models](https://mlsanity.com/paper/2209.04683/) (2022)
- [A Globally Convergent Gradient-based Bilevel Hyperparameter Optimization Method](https://mlsanity.com/paper/2208.12118/) (2022)
- [When Losses Align: Gradient-Based Composite Loss Weighting for Efficient Pretraining](https://mlsanity.com/paper/2605.07756/) (2026)

## Abstract (arXiv)

The performance of deep neural networks is well-known to be sensitive to the setting of their hyperparameters. Recent advances in reverse-mode automatic differentiation allow for optimizing hyperparameters with gradients. The standard way of computing these gradients involves a forward and backward pass of computations. However, the backward pass usually needs to consume unaffordable memory to store all the intermediate variables to exactly reverse the forward training procedure. In this work we propose a simple but effective method, DrMAD, to distill the knowledge of the forward pass into a shortcut path, through which we approximately reverse the training trajectory. Experiments on several image benchmark datasets show that DrMAD is at least 45 times faster and consumes 100 times less memory compared to state-of-the-art methods for optimizing hyperparameters with minimal compromise to its effectiveness. To the best of our knowledge, DrMAD is the first research attempt to make it practical to automatically tune thousands of hyperparameters of deep neural networks. The code can be downloaded from https://github.com/bigaidream-projects/drmad