# Regularized PolyKervNets: Optimizing Expressiveness and Efficiency for Private Inference in Deep Neural Networks

arXiv: https://arxiv.org/abs/2312.15229 · HTML: https://mlsanity.com/paper/2312.15229/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 30 references that are in the map.
- First posted on arXiv 23 Dec 2023.

## Builds on (most cited first, 19 of 19)

- [Gazelle: A Low Latency Framework for Secure Neural Network Inference](https://mlsanity.com/paper/1801.05507/) (2018)
- [CrypTen: Secure Multi-Party Computation Meets Machine Learning](https://mlsanity.com/paper/2109.00984/) (2021)
- [AdderNet: Do We Really Need Multiplications in Deep Learning?](https://mlsanity.com/paper/1912.13200/) (2019)
- [CrypTFlow2: Practical 2-Party Secure Inference](https://mlsanity.com/paper/2010.06457/) (2020)
- [MPCFormer: fast, performant and private Transformer inference with MPC](https://mlsanity.com/paper/2211.01452/) (2022)
- [XONN: XNOR-based Oblivious Deep Neural Network Inference](https://mlsanity.com/paper/1902.07342/) (2019)
- [DeepReDuce: ReLU Reduction for Fast Private Inference](https://mlsanity.com/paper/2103.01396/) (2021)
- [CryptoNAS: Private Inference on a ReLU Budget](https://mlsanity.com/paper/2006.08733/) (2020)
- [AutoReP: Automatic ReLU Replacement for Fast Private Network Inference](https://mlsanity.com/paper/2308.10134/) (2023)
- [MoMo: Momentum Models for Adaptive Learning Rates](https://mlsanity.com/paper/2305.07583/) (2023)
- [Kervolutional Neural Networks](https://mlsanity.com/paper/1904.03955/) (2019)
- [On Polynomial Approximations for Privacy-Preserving and Verifiable ReLU Networks](https://mlsanity.com/paper/2011.05530/) (2020)
- [Sisyphus: A Cautionary Tale of Using Low-Degree Polynomial Activations in Privacy-Preserving Deep Learning](https://mlsanity.com/paper/2107.12342/) (2021)
- [AESPA: Accuracy Preserving Low-degree Polynomial Activation for Fast Private Inference](https://mlsanity.com/paper/2201.06699/) (2022)
- [DeepReShape: Redesigning Neural Networks for Efficient Private Inference](https://mlsanity.com/paper/2304.10593/) (2023)
- [Fast and Private Inference of Deep Neural Networks by Co-designing Activation Functions](https://mlsanity.com/paper/2306.08538/) (2023)
- [SSIVD-Net: A Novel Salient Super Image Classification & Detection Technique for Weaponized Violence](https://mlsanity.com/paper/2207.12850/) (2022)
- [Kernel function impact on convolutional neural networks](https://mlsanity.com/paper/2302.10266/) (2023)
- [Compact: Approximating Complex Activation Functions for Secure Computation](https://mlsanity.com/paper/2309.04664/) (2023)

## Abstract (arXiv)

Private computation of nonlinear functions, such as Rectified Linear Units (ReLUs) and max-pooling operations, in deep neural networks (DNNs) poses significant challenges in terms of storage, bandwidth, and time consumption. To address these challenges, there has been a growing interest in utilizing privacy-preserving techniques that leverage polynomial activation functions and kernelized convolutions as alternatives to traditional ReLUs. However, these alternative approaches often suffer from a trade-off between achieving faster private inference (PI) and sacrificing model accuracy. In particular, when applied to much deeper networks, these methods encounter training instabilities, leading to issues like exploding gradients (resulting in NaNs) or suboptimal approximations. In this study, we focus on PolyKervNets, a technique known for offering improved dynamic approximations in smaller networks but still facing instabilities in larger and more complex networks. Our primary objective is to empirically explore optimization-based training recipes to enhance the performance of PolyKervNets in larger networks. By doing so, we aim to potentially eliminate the need for traditional nonlinear activation functions, thereby advancing the state-of-the-art in privacy-preserving deep neural network architectures. Code can be found on GitHub at: urlhttps://github.com/tolusophy/PolyKervNets/