# Neural Path Features and Neural Path Kernel : Understanding the role of gates in deep learning

arXiv: https://arxiv.org/abs/2006.10529 · HTML: https://mlsanity.com/paper/2006.10529/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jun 2020.

## Built on by (most cited first, 8 of 8)

- [Nonlinear Advantage: Trained Networks Might Not Be As Complex as You Think](https://mlsanity.com/paper/2211.17180/) (2022)
- [The Implicit Bias of Gradient Descent on Generalized Gated Linear Networks](https://mlsanity.com/paper/2202.02649/) (2022)
- [Disentangling deep neural networks with rectified linear units using duality](https://mlsanity.com/paper/2110.03403/) (2021)
- [Explicitising The Implicit Intrepretability of Deep Neural Networks Via Duality](https://mlsanity.com/paper/2203.16455/) (2022)
- [Half-Space Feature Learning in Neural Networks](https://mlsanity.com/paper/2404.04312/) (2024)
- [Interpreting Adversarial Attacks and Defences using Architectures with Enhanced Interpretability](https://mlsanity.com/paper/2502.15017/) (2025)
- [Pulling Back the Curtain on Deep Networks](https://mlsanity.com/paper/2507.22832/) (2025)
- [Hidden Activations are not Enough I: Knowledge Matrices as Higher Representations](https://mlsanity.com/paper/2609.34166/) (2026)

## Abstract (arXiv)

Rectified linear unit (ReLU) activations can also be thought of as 'gates', which, either pass or stop their pre-activation input when they are 'on' (when the pre-activation input is positive) or 'off' (when the pre-activation input is negative) respectively. A deep neural network (DNN) with ReLU activations has many gates, and the on/off status of each gate changes across input examples as well as network weights. For a given input example, only a subset of gates are 'active', i.e., on, and the sub-network of weights connected to these active gates is responsible for producing the output. At randomised initialisation, the active sub-network corresponding to a given input example is random. During training, as the weights are learnt, the active sub-networks are also learnt, and potentially hold very valuable information. In this paper, we analytically characterise the role of active sub-networks in deep learning. To this end, we encode the on/off state of the gates of a given input in a novel 'neural path feature' (NPF), and the weights of the DNN are encoded in a novel 'neural path value' (NPV). Further, we show that the output of network is indeed the inner product of NPF and NPV. The main result of the paper shows that the 'neural path kernel' associated with the NPF is a fundamental quantity that characterises the information stored in the gates of a DNN. We show via experiments (on MNIST and CIFAR-10) that in standard DNNs with ReLU activations NPFs are learnt during training and such learning is key for generalisation. Furthermore, NPFs and NPVs can be learnt in two separate networks and such learning also generalises well in experiments.