# Fast Jacobian-Vector Product for Deep Networks

arXiv: https://arxiv.org/abs/2104.00219 · HTML: https://mlsanity.com/paper/2104.00219/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 76 references that are in the map.
- First posted on arXiv 1 Apr 2021.

## Builds on (most cited first, 30 of 32)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [Automatic differentiation in machine learning: a survey](https://mlsanity.com/paper/1502.05767/) (2015)
- [Cyclical Learning Rates for Training Neural Networks](https://mlsanity.com/paper/1506.01186/) (2015)
- [Network In Network](https://mlsanity.com/paper/1312.4400/) (2013)
- [On the Number of Linear Regions of Deep Neural Networks](https://mlsanity.com/paper/1402.1869/) (2014)
- [The Effectiveness of Data Augmentation in Image Classification using Deep Learning](https://mlsanity.com/paper/1712.04621/) (2017)
- [Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks](https://mlsanity.com/paper/1703.02930/) (2017)
- [Understanding Deep Neural Networks with Rectified Linear Units](https://mlsanity.com/paper/1611.01491/) (2016)
- [Deep Learning: A Critical Appraisal](https://mlsanity.com/paper/1801.00631/) (2018)
- [Complexity of Linear Regions in Deep Networks](https://mlsanity.com/paper/1901.09021/) (2019)
- [Bounding and Counting Linear Regions of Deep Neural Networks](https://mlsanity.com/paper/1711.02114/) (2017)
- [Orthogonal Convolutional Neural Networks](https://mlsanity.com/paper/1911.12207/) (2019)
- [On the number of response regions of deep feed forward networks with piece-wise linear activations](https://mlsanity.com/paper/1312.6098/) (2013)
- [Orthogonal Deep Neural Networks](https://mlsanity.com/paper/1905.05929/) (2019)
- [Data augmentation instead of explicit regularization](https://mlsanity.com/paper/1806.03852/) (2018)
- [Mad Max: Affine Spline Insights into Deep Learning](https://mlsanity.com/paper/1805.06576/) (2018)
- [The Geometry of Deep Networks: Power Diagram Subdivision](https://mlsanity.com/paper/1905.08443/) (2019)
- [Iterative energy-based projection on a normal data manifold for anomaly localization](https://mlsanity.com/paper/2002.03734/) (2020)
- [Imposing Hard Constraints on Deep Networks: Promises and Limitations](https://mlsanity.com/paper/1706.02025/) (2017)
- [A representer theorem for deep neural networks](https://mlsanity.com/paper/1802.09210/) (2018)
- [Improving Deep Learning using Generic Data Augmentation](https://mlsanity.com/paper/1708.06020/) (2017)
- [On transversality of bent hyperplane arrangements and the topological expressiveness of ReLU neural networks](https://mlsanity.com/paper/2008.09052/) (2020)
- [AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks](https://mlsanity.com/paper/1712.02029/) (2017)
- [Learning disconnected manifolds: a no GANs land](https://mlsanity.com/paper/2006.04596/) (2020)
- [The simple essence of automatic differentiation](https://mlsanity.com/paper/1804.00746/) (2018)
- [Max-Affine Spline Insights into Deep Generative Networks](https://mlsanity.com/paper/2002.11912/) (2020)
- [Expression of Fractals Through Neural Network Functions](https://mlsanity.com/paper/1905.11345/) (2019)
- [Piecewise convexity of artificial neural networks](https://mlsanity.com/paper/1607.04917/) (2016)

## Built on by (most cited first, 4 of 4)

- [On the adaptation of recurrent neural networks for system identification](https://mlsanity.com/paper/2201.08660/) (2022)
- [Equivalent Linear Mappings of Large Language Models](https://mlsanity.com/paper/2505.24293/) (2025)
- [GrokAlign: Geometric Characterisation and Acceleration of Grokking](https://mlsanity.com/paper/2506.12284/) (2025)
- [Adjoint sharding for very long context training of state space models](https://mlsanity.com/paper/2501.00692/) (2025)

## Abstract (arXiv)

Jacobian-vector products (JVPs) form the backbone of many recent developments in Deep Networks (DNs), with applications including faster constrained optimization, regularization with generalization guarantees, and adversarial example sensitivity assessments. Unfortunately, JVPs are computationally expensive for real world DN architectures and require the use of automatic differentiation to avoid manually adapting the JVP program when changing the DN architecture. We propose a novel method to quickly compute JVPs for any DN that employ Continuous Piecewise Affine (e.g., leaky-ReLU, max-pooling, maxout, etc.) nonlinearities. We show that our technique is on average 2× faster than the fastest alternative over 13 DN architectures and across various hardware. In addition, our solution does not require automatic differentiation and is thus easy to deploy in software, requiring only the modification of a few lines of codes that do not depend on the DN architecture.