# On Polynomial Approximations for Privacy-Preserving and Verifiable ReLU Networks

arXiv: https://arxiv.org/abs/2011.05530 · HTML: https://mlsanity.com/paper/2011.05530/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Nov 2020.

## Built on by (most cited first, 19 of 19)

- [Stability and Generalization of Bilevel Programming in Hyperparameter Optimization](https://mlsanity.com/paper/2106.04188/) (2021)
- [AutoReP: Automatic ReLU Replacement for Fast Private Network Inference](https://mlsanity.com/paper/2308.10134/) (2023)
- [LinGCN: Structural Linearized Graph Convolutional Network for Homomorphically Encrypted Inference](https://mlsanity.com/paper/2309.14331/) (2023)
- [A Vertical Federated Learning Framework for Graph Convolutional Network](https://mlsanity.com/paper/2106.11593/) (2021)
- [zkDL: Efficient Zero-Knowledge Proofs of Deep Learning Training](https://mlsanity.com/paper/2307.16273/) (2023)
- [Fully Homomorphically Encrypted Deep Learning as a Service](https://mlsanity.com/paper/2107.12997/) (2021)
- [ApproxIFER: A Model-Agnostic Approach to Resilient and Robust Prediction Serving Systems](https://mlsanity.com/paper/2109.09868/) (2021)
- [On Regularizing Coordinate-MLPs](https://mlsanity.com/paper/2202.00790/) (2022)
- [Proof-of-Contribution-Based Design for Collaborative Machine Learning on Blockchain](https://mlsanity.com/paper/2302.14031/) (2023)
- [Adaptive Verifiable Coded Computing: Towards Fast, Secure and Private Distributed Machine Learning](https://mlsanity.com/paper/2107.12958/) (2021)
- [Optimized Layerwise Approximation for Efficient Private Inference on Fully Homomorphic Encryption](https://mlsanity.com/paper/2310.10349/) (2023)
- [All Rivers Run to the Sea: Private Learning with Asymmetric Flows](https://mlsanity.com/paper/2312.05264/) (2023)
- [A Training Framework for Optimal and Stable Training of Polynomial Neural Networks](https://mlsanity.com/paper/2505.11589/) (2025)
- [PolyMPCNet: Towards ReLU-free Neural Architecture Search in Two-party Computation Based Private Inference](https://mlsanity.com/paper/2209.09424/) (2022)
- [On the Fourier analysis in the SO(3) space : EquiLoPO Network](https://mlsanity.com/paper/2404.15979/) (2024)
- [ReBoot: Encrypted Training of Deep Neural Networks with CKKS Bootstrapping](https://mlsanity.com/paper/2506.19693/) (2025)
- [Lipschitz Multiscale Deep Equilibrium Models: A Theoretically Guaranteed and Accelerated Approach](https://mlsanity.com/paper/2602.03297/) (2026)
- [Regularized PolyKervNets: Optimizing Expressiveness and Efficiency for Private Inference in Deep Neural Networks](https://mlsanity.com/paper/2312.15229/) (2023)
- [Robust and Secure Code Watermarking for Large Language Models via ML/Crypto Codesign](https://mlsanity.com/paper/2502.02068/) (2025)

## Abstract (arXiv)

Outsourcing deep neural networks (DNNs) inference tasks to an untrusted cloud raises data privacy and integrity concerns. While there are many techniques to ensure privacy and integrity for polynomial-based computations, DNNs involve non-polynomial computations. To address these challenges, several privacy-preserving and verifiable inference techniques have been proposed based on replacing the non-polynomial activation functions such as the rectified linear unit (ReLU) function with polynomial activation functions. Such techniques usually require polynomials with integer coefficients or polynomials over finite fields. Motivated by such requirements, several works proposed replacing the ReLU function with the square function. In this work, we empirically show that the square function is not the best degree-2 polynomial that can replace the ReLU function even when restricting the polynomials to have integer coefficients. We instead propose a degree-2 polynomial activation function with a first order term and empirically show that it can lead to much better models. Our experiments on the CIFAR and Tiny ImageNet datasets on various architectures such as VGG-16 show that our proposed function improves the test accuracy by up to 10.4% compared to the square function.