# Plug-in, Trainable Gate for Streamlining Arbitrary Neural Networks

arXiv: https://arxiv.org/abs/1904.10921 · HTML: https://mlsanity.com/paper/1904.10921/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Apr 2019.

## Built on by (most cited first, 6 of 6)

- [Auto-Train-Once: Controller Network Guided Automatic Network Pruning from Scratch](https://mlsanity.com/paper/2403.14729/) (2024)
- [Injecting Logical Constraints into Neural Networks via Straight-Through Estimators](https://mlsanity.com/paper/2307.04347/) (2023)
- [CFDP: Common Frequency Domain Pruning](https://mlsanity.com/paper/2306.04147/) (2023)
- [Auto-Compressing Subset Pruning for Semantic Image Segmentation](https://mlsanity.com/paper/2201.11103/) (2022)
- [SpiderNet: Hybrid Differentiable-Evolutionary Architecture Search via Train-Free Metrics](https://mlsanity.com/paper/2204.09320/) (2022)
- [CWP: Instance complexity weighted channel-wise soft masks for network pruning](https://mlsanity.com/paper/2209.03534/) (2022)

## Abstract (arXiv)

Architecture optimization, which is a technique for finding an efficient neural network that meets certain requirements, generally reduces to a set of multiple-choice selection problems among alternative sub-structures or parameters. The discrete nature of the selection problem, however, makes this optimization difficult. To tackle this problem we introduce a novel concept of a trainable gate function. The trainable gate function, which confers a differentiable property to discretevalued variables, allows us to directly optimize loss functions that include non-differentiable discrete values such as 0-1 selection. The proposed trainable gate can be applied to pruning. Pruning can be carried out simply by appending the proposed trainable gate functions to each intermediate output tensor followed by fine-tuning the overall model, using any gradient-based training methods. So the proposed method can jointly optimize the selection of the pruned channels while fine-tuning the weights of the pruned model at the same time. Our experimental results demonstrate that the proposed method efficiently optimizes arbitrary neural networks in various tasks such as image classification, style transfer, optical flow estimation, and neural machine translation.