mlsanityData to 7 Oct 2026

Learning to Linearize Deep Neural Networks for Secure and Efficient Private Inference

arXiv 2301.09254 ↗First posted 23 Jan 2023cs.CV cs.AI cs.LGcited by 20

Not on the map. Why?

Abstract (arXiv)

The large number of ReLU non-linearity operations in existing deep neural networks makes them ill-suited for latency-efficient private inference (PI). Existing techniques to reduce ReLU operations often involve manual effort and sacrifice significant accuracy. In this paper, we first present a novel measure of non-linearity layers' ReLU sensitivity, enabling mitigation of the time-consuming manual efforts in identifying the same. Based on this sensitivity, we then present SENet, a three-stage training method that for a given ReLU budget, automatically assigns per-layer ReLU counts, decides the ReLU locations for each layer's activation map, and trains a model with significantly fewer ReLUs to potentially yield latency and communication efficient PI. Experimental evaluations with multiple models on various datasets show SENet's superior performance both in terms of reduced ReLUs and improved classification accuracy compared to existing alternatives. In particular, SENet can yield models that require up to ~2x fewer ReLUs while yielding similar accuracy. For a similar ReLU budget SENet can yield models with ~2.32% improved classification accuracy, evaluated on CIFAR-100.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 27 of its 45 references that are in the map. First posted on arXiv 23 Jan 2023.

Builds on

27 of 45 references.

References · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2016Wide Residual Networkscited by 3,057
  5. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  6. 2017A Survey on Deep Learning in Medical Image Analysiscited by 1,182
  7. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  8. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  9. 2020Fast is better than free: Revisiting adversarial trainingcited by 723
  10. 2018Rethinking the Value of Network Pruningcited by 595
  11. 2019Dreaming to Distill: Data-free Knowledge Transfer via DeepInversioncited by 419
  12. 2018Gazelle: A Low Latency Framework for Secure Neural Network Inferencecited by 193
15 more
  1. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  2. 2020Soft Threshold Weight Reparameterization for Learnable Sparsitycited by 150
  3. 2019Winning the Lottery with Continuous Sparsificationcited by 99
  4. 2019Global Sparse Momentum SGD for Pruning Very Deep Neural Networkscited by 70
  5. 2021DeepReDuce: ReLU Reduction for Fast Private Inferencecited by 50
  6. 2019AutoCompress: An Automatic DNN Structured Pruning Framework for Ultra-High Compression Ratescited by 49
  7. 2020Once-for-All Adversarial Training: In-Situ Tradeoff between Robustness and Accuracy for Freecited by 48
  8. 2020CryptoNAS: Private Inference on a ReLU Budgetcited by 40
  9. 2020A Tunable Robust Pruning Framework Through Dynamic Network Rewiring of DNNscited by 37
  10. 2022Selective Network Linearization for Efficient Private Inferencecited by 31
  11. 2021Circa: Stochastic ReLUs for Private Deep Learningcited by 16
  12. 2020Pre-defined Sparsity for Low-Complexity Convolutional Neural Networkscited by 15
  13. 2020AttentionLite: Towards Efficient Self-Attention Models for Visioncited by 10
  14. 2021BMPQ: Bit-Gradient Sensitivity Driven Mixed-Precision Quantization of DNNs from Scratchcited by 10
  15. 2022A Fast and Efficient Conditional Learning for Tunable Trade-Off between Accuracy and Robustnesscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 10; 2025: 3; 2026: 2.

20 papers.

Sort:

By area:

  1. 2024The RoboDrive Challenge: Drive Anytime Anywhere in Any Conditioncited by 23
  2. 2024AdaPI: Facilitating DNN Model Adaptivity for Efficient Private Inference in Edge Computingcited by 15
  3. 2024PrivCirNet: Efficient Private Inference via Block Circulant Transformationcited by 7
  4. 2024MPC-Minimized Secure LLM Inferencecited by 5
  5. 2023Making Models Shallow Again: Jointly Learning to Reduce Non-Linearity and Depth for Latency-Efficient Private Inferencecited by 4
  6. 2023PriViT: Vision Transformers for Fast Private Inferencecited by 4
  7. 2024Block Selective Reprogramming for On-device Training of Vision Transformerscited by 4
  8. 2023Recent Advances in Scalable Energy-Efficient and Trustworthy Spiking Neural networks: from Algorithms to Technologycited by 3
  9. 2023Sensi-BERT: Towards Sensitivity Driven Fine-Tuning for Parameter-Efficient BERTcited by 2
  10. 2023Approximating ReLU on a Reduced Ring for Efficient MPC-based Private Inferencecited by 2
  11. 2024HEQuant: Marrying Homomorphic Encryption and Quantization for Communication-Efficient Private Inferencecited by 2
  12. 2024Disparate Impact on Group Accuracy of Linearization for Private Inferencecited by 2
8 more
  1. 2024TruncFormer: Private LLM Inference Using Only Truncationscited by 1
  2. 2025CrypTorch: PyTorch-based Auto-tuning Compiler for Machine Learning with Multi-party Computationcited by 1
  3. 2024Linearizing Models for Efficient yet Robust Private Inferencecited by 0
  4. 2024xMLP: Revolutionizing Private Inference with Exclusive Square Activationcited by 0
  5. 2025Coordinate Descent for Network Linearizationcited by 0
  6. 2025DeepShare: Sharing ReLU Across Channels and Layers for Efficient Private Inferencecited by 0
  7. 2026LRD-MPC: Efficient MPC Inference through Low-rank Decompositioncited by 0
  8. 2026UFO: Unlocking Ultra-Efficient Quantized Private Inference with Protocol and Algorithm Co-Optimizationcited by 0