# A Closer Look at Double Backpropagation

arXiv: https://arxiv.org/abs/1906.06637 · HTML: https://mlsanity.com/paper/1906.06637/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jun 2019.

## Built on by (most cited first, 7 of 7)

- [Regulating Model Reliance on Non-Robust Features by Smoothing Input Marginal Density](https://mlsanity.com/paper/2407.04370/) (2024)
- [Efficient local linearity regularization to overcome catastrophic overfitting](https://mlsanity.com/paper/2401.11618/) (2024)
- [Sequence Modeling with Spectral Mean Flows](https://mlsanity.com/paper/2510.15366/) (2025)
- [Goal-Conditioned Q-Learning as Knowledge Distillation](https://mlsanity.com/paper/2208.13298/) (2022)
- [Layer-wise Derivative Controlled Networks](https://mlsanity.com/paper/2605.15463/) (2026)
- [Model Guidance via Explanations Turns Image Classifiers into Segmentation Models](https://mlsanity.com/paper/2407.03009/) (2024)
- [DReS: Dual Reconstruction Smoothing for Functional Regularization](https://mlsanity.com/paper/2510.00253/) (2025)

## Abstract (arXiv)

In recent years, an increasing number of neural network models have included derivatives with respect to inputs in their loss functions, resulting in so-called double backpropagation for first-order optimization. However, so far no general description of the involved derivatives exists. Here, we cover a wide array of special cases in a very general Hilbert space framework, which allows us to provide optimized backpropagation rules for many real-world scenarios. This includes the reduction of calculations for Frobenius-norm-penalties on Jacobians by roughly a third for locally linear activation functions. Furthermore, we provide a description of the discontinuous loss surface of ReLU networks both in the inputs and the parameters and demonstrate why the discontinuities do not pose a big problem in reality.