mlsanityData to 14 Oct 2026

AutoReP: Automatic ReLU Replacement for Fast Private Network Inference

arXiv 2308.10134 ↗First posted 20 Aug 2023cs.CR cs.LGcited by 29

Not on the map. Why?

Abstract (arXiv)

The growth of the Machine-Learning-As-A-Service (MLaaS) market has highlighted clients' data privacy and security issues. Private inference (PI) techniques using cryptographic primitives offer a solution but often have high computation and communication costs, particularly with non-linear operators like ReLU. Many attempts to reduce ReLU operations exist, but they may need heuristic threshold selection or cause substantial accuracy loss. This work introduces AutoReP, a gradient-based approach to lessen non-linear operators and alleviate these issues. It automates the selection of ReLU and polynomial functions to speed up PI applications and introduces distribution-aware polynomial approximation (DaPa) to maintain model expressivity while accurately approximating ReLUs. Our experimental results demonstrate significant accuracy improvements of 6.12% (94.31%, 12.9K ReLU budget, CIFAR-10), 8.39% (74.92%, 12.9K ReLU budget, CIFAR-100), and 9.45% (63.69%, 55K ReLU budget, Tiny-ImageNet) over current state-of-the-art methods, e.g., SNL. Morever, AutoReP is applied to EfficientNet-B2 on ImageNet dataset, and achieved 75.55% accuracy with 176.1 times ReLU budget reduction.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Builds on 51 of its 90 references that are in the map. First posted on arXiv 20 Aug 2023.

Builds on

51 of 90 references.

References · 51

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  3. 2016Wide Residual Networkscited by 3,070
  4. 2016Pruning Filters for Efficient ConvNetscited by 1,143
  5. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
  6. 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 377
  7. 2016Dynamic Network Surgery for Efficient DNNscited by 233
  8. 2019Understanding Straight-Through Estimator in Training Activation Quantized Neural Netscited by 231
  9. 2018Gazelle: A Low Latency Framework for Secure Neural Network Inferencecited by 193
  10. 2021CrypTen: Secure Multi-Party Computation Meets Machine Learningcited by 131
  11. 2022Brain Network Transformercited by 94
  12. 2022What Makes Convolutional Models Great on Long Sequence Modeling?cited by 87
38 more
  1. 2016Training Sparse Neural Networkscited by 73
  2. 2022FBNETGEN: Task-aware GNN-based fMRI Analysis via Functional Brain Network Generationcited by 67
  3. 2022Accelerating Dataset Distillation via Model Augmentationcited by 62
  4. 2018Low Latency Privacy Preserving Inferencecited by 57
  5. 2019XONN: XNOR-based Oblivious Deep Neural Network Inferencecited by 51
  6. 2021DeepReDuce: ReLU Reduction for Fast Private Inferencecited by 50
  7. 2020CryptoNAS: Private Inference on a ReLU Budgetcited by 40
  8. 2022Federated Contrastive Learning for Volumetric Medical Image Segmentationcited by 33
  9. 2020Against Membership Inference Attack: Pruning is All You Needcited by 32
  10. 2022Selective Network Linearization for Efficient Private Inferencecited by 31
  11. 2021Accelerating Framework of Transformer by Hardware Design and Model Compression Co-Optimizationcited by 27
  12. 2020Fast OSCAR and OWL Regression via Safe Screening Rulescited by 25
  13. 2020Enabling On-Device CNN Training by Self-Supervised Instance Filtering and Error Map Pruningcited by 24
  14. 2020Training Binary Neural Networks using the Bayesian Learning Rulecited by 23
  15. 2022A Length Adaptive Algorithm-Hardware Co-design of Transformer on FPGA Through Sparse Attention and Dynamic Pipeliningcited by 22
  16. 2021Enabling On-Device Self-Supervised Contrastive Learning With Selective Data Contrastcited by 21
  17. 2021Zero-Shot Scene Graph Relation Prediction through Commonsense Knowledge Integrationcited by 21
  18. 2021Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigmcited by 21
  19. 2022Federated Contrastive Learning for Dermatological Disease Diagnosis via On-device Learningcited by 21
  20. 2022The Larger The Fairer? Small Neural Networks Can Achieve Fairness for Edge Devicescited by 21
  21. 2020Intermittent Inference with Nonuniformly Compressed Multi-Exit Neural Network for Energy Harvesting Powered Devicescited by 19
  22. 2020On Polynomial Approximations for Privacy-Preserving and Verifiable ReLU Networkscited by 19
  23. 2022Towards Sparsification of Graph Neural Networkscited by 18
  24. 2022Distributed Contrastive Learning for Medical Image Segmentationcited by 16
  25. 2021Sphynx: ReLU-Efficient Network Design for Private Inferencecited by 15
  26. 2021Decentralized Unsupervised Learning of Visual Representationscited by 15
  27. 2022Dynamic Sparse Training via Balancing the Exploration-Exploitation Trade-offcited by 15
  28. 2018TAPAS: Tricks to Accelerate (encrypted) Prediction As a Servicecited by 14
  29. 2021Sisyphus: A Cautionary Tale of Using Low-Degree Polynomial Activations in Privacy-Preserving Deep Learningcited by 14
  30. 2021Generic Neural Architecture Search via Regressioncited by 14
  31. 2021Binary Complex Neural Network Acceleration on FPGAcited by 14
  32. 2022An Accelerated Doubly Stochastic Gradient Method with Faster Explicit Model Identificationcited by 14
  33. 2023Neighborhood-Regularized Self-Training for Learning with Few Labelscited by 14
  34. 2022An Automatic and Efficient BERT Pruning for Edge AI Systemscited by 13
  35. 2022Synthetic Data Can Also Teach: Synthesizing Effective Data for Unsupervised Visual Representation Learningcited by 12
  36. 2021Secure Human Action Recognition by Encrypted Neural Network Inferencecited by 10
  37. 2023Neurogenesis Dynamics-inspired Spiking Neural Network Training Accelerationcited by 10
  38. 2019A Hybrid Approach to Secure Function Evaluation Using SGXcited by 6

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 19; 2025: 3; 2026: 1.

29 papers.

Sort:

By area:

  1. 2024Learning from Teaching Regularization: Generalizable Correlations Should be Easy to Imitatecited by 47
  2. 2024Time Series Modeling for Heart Rate Prediction: From ARIMA to Transformerscited by 35
  3. 2024Regional Style and Color Transfercited by 30
  4. 2024APEER: Automatic Prompt Engineering Enhances Large Language Model Rerankingcited by 28
  5. 2023LinGCN: Structural Linearized Graph Convolutional Network for Homomorphically Encrypted Inferencecited by 27
  6. 2023MaxK-GNN: Extremely Fast GPU Kernel Design for Accelerating Graph Neural Networks Trainingcited by 25
  7. 2024A Comparative Study on Enhancing Prediction in Social Network Advertisement through Data Augmentationcited by 21
  8. 2024Enhance Image-to-Image Generation with LLaVA-generated Promptscited by 20
  9. 2024Cross-border Commodity Pricing Strategy Optimization via Mixed Neural Network for Time Series Analysiscited by 12
  10. 2024Deep Analysis of Time Series Data for Smart Grid Startup Strategies: A Transformer-LSTM-PSO Model Approachcited by 11
  11. 2024Advanced AI Framework for Enhanced Detection and Assessment of Abdominal Trauma: Integrating 3D Segmentation with 2D CNN and RNN Modelscited by 9
  12. 2023Muffin: A Framework Toward Multi-Dimension AI Fairness by Uniting Off-the-Shelf Modelscited by 8
17 more
  1. 2024DLoRA: Distributed Parameter-Efficient Fine-Tuning Solution for Large Language Modelcited by 8
  2. 2024Recording Brain Activity While Listening to Music Using Wearable EEG Devices Combined with Bidirectional Long Short-Term Memory Networkscited by 8
  3. 2024Privacy-Preserving Diffusion Model Using Homomorphic Encryptioncited by 5
  4. 2024PristiQ: A Co-Design Framework for Preserving Data Security of Quantum Learning in the Cloudcited by 5
  5. 2023Optimized Layerwise Approximation for Efficient Private Inference on Fully Homomorphic Encryptioncited by 4
  6. 2025Intelligent logistics management robot path planning algorithm integrating transformer and GCN networkcited by 4
  7. 2023Edge-InversionNet: Enabling Efficient Inference of InversionNet on Edge Devicescited by 2
  8. 2024Data-Algorithm-Architecture Co-Optimization for Fair Neural Networks on Skin Lesion Datasetcited by 2
  9. 2024Key Information Retrieval to Classify the Unstructured Data Content of Preferential Trade Agreementscited by 1
  10. 2024Neural Networks with (Low-Precision) Polynomial Approximations: New Insights and Techniques for Accuracy Improvementcited by 1
  11. 2024A Self-guided Multimodal Approach to Enhancing Graph Representation Learning for Alzheimer's Diseasescited by 1
  12. 2023Regularized PolyKervNets: Optimizing Expressiveness and Efficiency for Private Inference in Deep Neural Networkscited by 0
  13. 2024International Trade Flow Prediction with Bilateral Trade Provisionscited by 0
  14. 2024Risk Analysis in Customer Relationship Management via Quantile Region Convolutional Neural Network-Long Short-Term Memory and Cross-Attention Mechanismcited by 0
  15. 2025Coordinate Descent for Network Linearizationcited by 0
  16. 2025DeepShare: Sharing ReLU Across Channels and Layers for Efficient Private Inferencecited by 0
  17. 2026UFO: Unlocking Ultra-Efficient Quantized Private Inference with Protocol and Algorithm Co-Optimizationcited by 0