mlsanityData to 7 Oct 2026

Making Models Shallow Again: Jointly Learning to Reduce Non-Linearity and Depth for Latency-Efficient Private Inference

arXiv 2304.13274 ↗First posted 26 Apr 2023cs.LG cs.CRcited by 4

Not on the map. Why?

Abstract (arXiv)

Large number of ReLU and MAC operations of Deep neural networks make them ill-suited for latency and compute-efficient private inference. In this paper, we present a model optimization method that allows a model to learn to be shallow. In particular, we leverage the ReLU sensitivity of a convolutional block to remove a ReLU layer and merge its succeeding and preceding convolution layers to a shallow block. Unlike existing ReLU reduction methods, our joint reduction method can yield models with improved reduction of both ReLUs and linear operations by up to 1.73x and 1.47x, respectively, evaluated with ResNet18 on CIFAR-100 without any significant accuracy-drop.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 14 of its 26 references that are in the map. First posted on arXiv 26 Apr 2023.

Builds on

14 of 26 references.

References · 14

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016Wide Residual Networkscited by 3,057
  3. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 491
  4. 2018Gazelle: A Low Latency Framework for Secure Neural Network Inferencecited by 193
  5. 2021DeepReDuce: ReLU Reduction for Fast Private Inferencecited by 50
  6. 2021CryptGPU: Fast Privacy-Preserving Machine Learning on the GPUcited by 46
  7. 2020CryptoNAS: Private Inference on a ReLU Budgetcited by 40
  8. 2020A Tunable Robust Pruning Framework Through Dynamic Network Rewiring of DNNscited by 37
  9. 2022Selective Network Linearization for Efficient Private Inferencecited by 31
  10. 2023Learning to Linearize Deep Neural Networks for Secure and Efficient Private Inferencecited by 20
  11. 2022DepthShrinker: A New Compression Paradigm Towards Boosting Real-Hardware Efficiency of Compact Neural Networkscited by 19
  12. 2021Sphynx: ReLU-Efficient Network Design for Private Inferencecited by 15
2 more
  1. 2020AttentionLite: Towards Efficient Self-Attention Models for Visioncited by 10
  2. 2022Sparse Mixture Once-for-all Adversarial Training for Efficient In-Situ Trade-Off Between Accuracy and Robustness of DNNscited by 4

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 2.

4 papers.

  1. 2023Mitigate Replication and Copying in Diffusion Models with Generalized Caption and Dual Fusion Enhancementcited by 5
  2. 2023FakeSwarm: Improving Fake News Detection with Swarming Characteristicscited by 1
  3. 2024Linearizing Models for Efficient yet Robust Private Inferencecited by 0
  4. 2024CiMNet: Towards Joint Optimization for DNN Architecture and Configuration for Compute-In-Memory Hardwarecited by 0