# Learning to Linearize Deep Neural Networks for Secure and Efficient Private Inference

arXiv: https://arxiv.org/abs/2301.09254 · HTML: https://mlsanity.com/paper/2301.09254/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 45 references that are in the map.
- First posted on arXiv 23 Jan 2023.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [A Survey on Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/1702.05747/) (2017)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [Fast is better than free: Revisiting adversarial training](https://mlsanity.com/paper/2001.03994/) (2020)
- [Rethinking the Value of Network Pruning](https://mlsanity.com/paper/1810.05270/) (2018)
- [Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion](https://mlsanity.com/paper/1912.08795/) (2019)
- [Gazelle: A Low Latency Framework for Secure Neural Network Inference](https://mlsanity.com/paper/1801.05507/) (2018)
- [FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout](https://mlsanity.com/paper/2102.13451/) (2021)
- [Soft Threshold Weight Reparameterization for Learnable Sparsity](https://mlsanity.com/paper/2002.03231/) (2020)
- [Winning the Lottery with Continuous Sparsification](https://mlsanity.com/paper/1912.04427/) (2019)
- [Global Sparse Momentum SGD for Pruning Very Deep Neural Networks](https://mlsanity.com/paper/1909.12778/) (2019)
- [DeepReDuce: ReLU Reduction for Fast Private Inference](https://mlsanity.com/paper/2103.01396/) (2021)
- [AutoCompress: An Automatic DNN Structured Pruning Framework for Ultra-High Compression Rates](https://mlsanity.com/paper/1907.03141/) (2019)
- [Once-for-All Adversarial Training: In-Situ Tradeoff between Robustness and Accuracy for Free](https://mlsanity.com/paper/2010.11828/) (2020)
- [CryptoNAS: Private Inference on a ReLU Budget](https://mlsanity.com/paper/2006.08733/) (2020)
- [A Tunable Robust Pruning Framework Through Dynamic Network Rewiring of DNNs](https://mlsanity.com/paper/2011.03083/) (2020)
- [Selective Network Linearization for Efficient Private Inference](https://mlsanity.com/paper/2202.02340/) (2022)
- [Circa: Stochastic ReLUs for Private Deep Learning](https://mlsanity.com/paper/2106.08475/) (2021)
- [Pre-defined Sparsity for Low-Complexity Convolutional Neural Networks](https://mlsanity.com/paper/2001.10710/) (2020)
- [BMPQ: Bit-Gradient Sensitivity Driven Mixed-Precision Quantization of DNNs from Scratch](https://mlsanity.com/paper/2112.13843/) (2021)
- [AttentionLite: Towards Efficient Self-Attention Models for Vision](https://mlsanity.com/paper/2101.05216/) (2020)
- [A Fast and Efficient Conditional Learning for Tunable Trade-Off between Accuracy and Robustness](https://mlsanity.com/paper/2204.00426/) (2022)

## Built on by (most cited first, 20 of 20)

- [The RoboDrive Challenge: Drive Anytime Anywhere in Any Condition](https://mlsanity.com/paper/2405.08816/) (2024)
- [AdaPI: Facilitating DNN Model Adaptivity for Efficient Private Inference in Edge Computing](https://mlsanity.com/paper/2407.05633/) (2024)
- [PrivCirNet: Efficient Private Inference via Block Circulant Transformation](https://mlsanity.com/paper/2405.14569/) (2024)
- [MPC-Minimized Secure LLM Inference](https://mlsanity.com/paper/2408.03561/) (2024)
- [Making Models Shallow Again: Jointly Learning to Reduce Non-Linearity and Depth for Latency-Efficient Private Inference](https://mlsanity.com/paper/2304.13274/) (2023)
- [PriViT: Vision Transformers for Fast Private Inference](https://mlsanity.com/paper/2310.04604/) (2023)
- [Block Selective Reprogramming for On-device Training of Vision Transformers](https://mlsanity.com/paper/2405.10951/) (2024)
- [Recent Advances in Scalable Energy-Efficient and Trustworthy Spiking Neural networks: from Algorithms to Technology](https://mlsanity.com/paper/2312.01213/) (2023)
- [Sensi-BERT: Towards Sensitivity Driven Fine-Tuning for Parameter-Efficient BERT](https://mlsanity.com/paper/2307.11764/) (2023)
- [Approximating ReLU on a Reduced Ring for Efficient MPC-based Private Inference](https://mlsanity.com/paper/2309.04875/) (2023)
- [HEQuant: Marrying Homomorphic Encryption and Quantization for Communication-Efficient Private Inference](https://mlsanity.com/paper/2401.15970/) (2024)
- [Disparate Impact on Group Accuracy of Linearization for Private Inference](https://mlsanity.com/paper/2402.03629/) (2024)
- [TruncFormer: Private LLM Inference Using Only Truncations](https://mlsanity.com/paper/2412.01042/) (2024)
- [CrypTorch: PyTorch-based Auto-tuning Compiler for Machine Learning with Multi-party Computation](https://mlsanity.com/paper/2511.19711/) (2025)
- [Linearizing Models for Efficient yet Robust Private Inference](https://mlsanity.com/paper/2402.05521/) (2024)
- [xMLP: Revolutionizing Private Inference with Exclusive Square Activation](https://mlsanity.com/paper/2403.08024/) (2024)
- [Coordinate Descent for Network Linearization](https://mlsanity.com/paper/2511.11781/) (2025)
- [DeepShare: Sharing ReLU Across Channels and Layers for Efficient Private Inference](https://mlsanity.com/paper/2512.17398/) (2025)
- [LRD-MPC: Efficient MPC Inference through Low-rank Decomposition](https://mlsanity.com/paper/2602.14397/) (2026)
- [UFO: Unlocking Ultra-Efficient Quantized Private Inference with Protocol and Algorithm Co-Optimization](https://mlsanity.com/paper/2602.18758/) (2026)

## Abstract (arXiv)

The large number of ReLU non-linearity operations in existing deep neural networks makes them ill-suited for latency-efficient private inference (PI). Existing techniques to reduce ReLU operations often involve manual effort and sacrifice significant accuracy. In this paper, we first present a novel measure of non-linearity layers' ReLU sensitivity, enabling mitigation of the time-consuming manual efforts in identifying the same. Based on this sensitivity, we then present SENet, a three-stage training method that for a given ReLU budget, automatically assigns per-layer ReLU counts, decides the ReLU locations for each layer's activation map, and trains a model with significantly fewer ReLUs to potentially yield latency and communication efficient PI. Experimental evaluations with multiple models on various datasets show SENet's superior performance both in terms of reduced ReLUs and improved classification accuracy compared to existing alternatives. In particular, SENet can yield models that require up to ~2x fewer ReLUs while yielding similar accuracy. For a similar ReLU budget SENet can yield models with ~2.32% improved classification accuracy, evaluated on CIFAR-100.