mlsanityData to 7 Oct 2026

DeepReDuce: ReLU Reduction for Fast Private Inference

arXiv 2103.01396 ↗First posted 2 Mar 2021cs.LG cs.CRcited by 50

Not on the map. Why?

Abstract (arXiv)

The recent rise of privacy concerns has led researchers to devise methods for private neural inference -- where inferences are made directly on encrypted data, never seeing inputs. The primary challenge facing private inference is that computing on encrypted data levies an impractically-high latency penalty, stemming mostly from non-linear operators like ReLU. Enabling practical and private inference requires new optimization methods that minimize network ReLU counts while preserving accuracy. This paper proposes DeepReDuce: a set of optimizations for the judicious removal of ReLUs to reduce private inference latency. The key insight is that not all ReLUs contribute equally to accuracy. We leverage this insight to drop, or remove, ReLUs from classic networks to significantly reduce inference latency and maintain high accuracy. Given a target network, DeepReDuce outputs a Pareto frontier of networks that tradeoff the number of ReLUs and accuracy. Compared to the state-of-the-art for private inference DeepReDuce improves accuracy and reduces ReLU count by up to 3.5% (iso-ReLU count) and 3.5× (iso-accuracy), respectively.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Builds on 17 of its 42 references that are in the map. First posted on arXiv 2 Mar 2021.

Builds on

17 of 42 references.

References · 17

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2016Densely Connected Convolutional Networkscited by 6,691
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  6. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  7. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  8. 2016Wide Residual Networkscited by 3,057
  9. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  10. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  11. 2019On the Efficacy of Knowledge Distillationcited by 377
  12. 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 312
5 more
  1. 2019Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNetcited by 260
  2. 2018Gazelle: A Low Latency Framework for Secure Neural Network Inferencecited by 193
  3. 2017CondenseNet: An Efficient DenseNet using Learned Group Convolutionscited by 121
  4. 2018A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillationcited by 110
  5. 2020CryptoNAS: Private Inference on a ReLU Budgetcited by 40

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 10; 2023: 16; 2024: 12; 2025: 3; 2026: 5.

50 papers.

Sort:

By area:

  1. 2022Selective Network Linearization for Efficient Private Inferencecited by 31
  2. 2022Scaling up Trustless DNN Inference with Zero-Knowledge Proofscited by 30
  3. 2023AutoReP: Automatic ReLU Replacement for Fast Private Network Inferencecited by 29
  4. 2023LinGCN: Structural Linearized Graph Convolutional Network for Homomorphically Encrypted Inferencecited by 26
  5. 2023Learning to Linearize Deep Neural Networks for Secure and Efficient Private Inferencecited by 20
  6. 2022MPCViT: Searching for Accurate and Efficient MPC-Friendly Vision Transformer with Heterogeneous Attentioncited by 19
  7. 2021Circa: Stochastic ReLUs for Private Deep Learningcited by 16
  8. 2021Sphynx: ReLU-Efficient Network Design for Private Inferencecited by 15
  9. 2023AutoFHE: Automated Adaption of CNNs for Efficient Evaluation over FHEcited by 15
  10. 2021Layer Folding: Neural Network Depth Reduction using Activation Linearizationcited by 14
  11. 2021Sisyphus: A Cautionary Tale of Using Low-Degree Polynomial Activations in Privacy-Preserving Deep Learningcited by 14
  12. 2022AESPA: Accuracy Preserving Low-degree Polynomial Activation for Fast Private Inferencecited by 14
38 more
  1. 2023RRNet: Towards ReLU-Reduced Neural Network for Two-party Computation Based Private Inferencecited by 12
  2. 2022CryptoGCN: Fast and Scalable Homomorphically Encrypted Graph Convolutional Network Inferencecited by 10
  3. 2024PrivCirNet: Efficient Private Inference via Block Circulant Transformationcited by 7
  4. 2022Private and Reliable Neural Network Inferencecited by 6
  5. 2023Training Large Scale Polynomial CNNs for E2E Inference over Homomorphic Encryptioncited by 5
  6. 2024MPC-Minimized Secure LLM Inferencecited by 5
  7. 2022MPC-Pipe: an Efficient Pipeline Scheme for Secure Multi-party Machine Learning Inferencecited by 4
  8. 2023Making Models Shallow Again: Jointly Learning to Reduce Non-Linearity and Depth for Latency-Efficient Private Inferencecited by 4
  9. 2023Fast and Private Inference of Deep Neural Networks by Co-designing Activation Functionscited by 4
  10. 2023PriViT: Vision Transformers for Fast Private Inferencecited by 4
  11. 2023Optimized Layerwise Approximation for Efficient Private Inference on Fully Homomorphic Encryptioncited by 4
  12. 2024Privacy-Preserving Diffusion Model Using Homomorphic Encryptioncited by 4
  13. 2024The Simpler The Better: An Entropy-Based Importance Metric To Reduce Neural Networks' Depthcited by 3
  14. 2024ReLU's Revival: On the Entropic Overload in Normalization-Free Large Language Modelscited by 3
  15. 2022Tabula: Efficiently Computing Nonlinear Activation Functions for Secure Neural Network Inferencecited by 2
  16. 2022Efficient ML Models for Practical Secure Inferencecited by 2
  17. 2022PolyMPCNet: Towards ReLU-free Neural Architecture Search in Two-party Computation Based Private Inferencecited by 2
  18. 2023Approximating ReLU on a Reduced Ring for Efficient MPC-based Private Inferencecited by 2
  19. 2024HEQuant: Marrying Homomorphic Encryption and Quantization for Communication-Efficient Private Inferencecited by 2
  20. 2024Disparate Impact on Group Accuracy of Linearization for Private Inferencecited by 2
  21. 2023Securing Neural Networks with Knapsack Optimizationcited by 1
  22. 2023NetBooster: Empowering Tiny Deep Learning By Standing on the Shoulders of Deep Giantscited by 1
  23. 2023Towards Fast and Scalable Private Inferencecited by 1
  24. 2023LayerCollapse: Adaptive compression of neural networkscited by 1
  25. 2024Neural Networks with (Low-Precision) Polynomial Approximations: New Insights and Techniques for Accuracy Improvementcited by 1
  26. 2024DCT-CryptoNets: Scaling Private Inference in the Frequency Domaincited by 1
  27. 2024PrivQuant: Communication-Efficient Private Inference with Quantized Network/Protocol Co-Optimizationcited by 1
  28. 2025CrypTorch: PyTorch-based Auto-tuning Compiler for Machine Learning with Multi-party Computationcited by 1
  29. 2026ATLAS: Automated Approximation of Transformers for Efficient Homomorphic Inference in One Hourcited by 1
  30. 2023Regularized PolyKervNets: Optimizing Expressiveness and Efficiency for Private Inference in Deep Neural Networkscited by 0
  31. 2024Linearizing Models for Efficient yet Robust Private Inferencecited by 0
  32. 2024xMLP: Revolutionizing Private Inference with Exclusive Square Activationcited by 0
  33. 2025Coordinate Descent for Network Linearizationcited by 0
  34. 2025DeepShare: Sharing ReLU Across Channels and Layers for Efficient Private Inferencecited by 0
  35. 2026UFO: Unlocking Ultra-Efficient Quantized Private Inference with Protocol and Algorithm Co-Optimizationcited by 0
  36. 2026Meeting in the Middle: A Co-Design Paradigm for FHE and AI Inferencecited by 0
  37. 2026PRISM: Sensitivity-Aware PolynoMial PRuning for EffIcient Neural Network Encryptioncited by 0
  38. 2026HEAT: Faster Fully Homomorphic Inference via Approximations-Weights Co-Adaptationcited by 0