# Circa: Stochastic ReLUs for Private Deep Learning

arXiv: https://arxiv.org/abs/2106.08475 · HTML: https://mlsanity.com/paper/2106.08475/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 30 references that are in the map.
- First posted on arXiv 15 Jun 2021.

## Builds on (most cited first, 15 of 15)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Gazelle: A Low Latency Framework for Secure Neural Network Inference](https://mlsanity.com/paper/1801.05507/) (2018)
- [Slalom: Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware](https://mlsanity.com/paper/1806.03287/) (2018)
- [FALCON: Honest-Majority Maliciously Secure Framework for Private Deep Learning](https://mlsanity.com/paper/2004.02229/) (2020)
- [XONN: XNOR-based Oblivious Deep Neural Network Inference](https://mlsanity.com/paper/1902.07342/) (2019)
- [DeepReDuce: ReLU Reduction for Fast Private Inference](https://mlsanity.com/paper/2103.01396/) (2021)
- [CryptGPU: Fast Privacy-Preserving Machine Learning on the GPU](https://mlsanity.com/paper/2104.10949/) (2021)
- [Faster CryptoNets: Leveraging Sparsity for Real-World Encrypted Inference](https://mlsanity.com/paper/1811.09953/) (2018)
- [CryptoNAS: Private Inference on a ReLU Budget](https://mlsanity.com/paper/2006.08733/) (2020)
- [ThUnderVolt: Enabling Aggressive Voltage Underscaling and Timing Error Resilience for Energy Efficient Deep Neural Network Accelerators](https://mlsanity.com/paper/1802.03806/) (2018)
- [MASR: A Modular Accelerator for Sparse RNNs](https://mlsanity.com/paper/1908.08976/) (2019)
- [Weightless: Lossy Weight Encoding For Deep Neural Network Compression](https://mlsanity.com/paper/1711.04686/) (2017)
- [NASS: Optimizing Secure Inference via Neural Architecture Search](https://mlsanity.com/paper/2001.11854/) (2020)

## Built on by (most cited first, 16 of 16)

- [Selective Network Linearization for Efficient Private Inference](https://mlsanity.com/paper/2202.02340/) (2022)
- [Learning to Linearize Deep Neural Networks for Secure and Efficient Private Inference](https://mlsanity.com/paper/2301.09254/) (2023)
- [AutoFHE: Automated Adaption of CNNs for Efficient Evaluation over FHE](https://mlsanity.com/paper/2310.08012/) (2023)
- [AESPA: Accuracy Preserving Low-degree Polynomial Activation for Fast Private Inference](https://mlsanity.com/paper/2201.06699/) (2022)
- [MPC-Pipe: an Efficient Pipeline Scheme for Secure Multi-party Machine Learning Inference](https://mlsanity.com/paper/2209.13643/) (2022)
- [PriViT: Vision Transformers for Fast Private Inference](https://mlsanity.com/paper/2310.04604/) (2023)
- [ReLU's Revival: On the Entropic Overload in Normalization-Free Large Language Models](https://mlsanity.com/paper/2410.09637/) (2024)
- [Tabula: Efficiently Computing Nonlinear Activation Functions for Secure Neural Network Inference](https://mlsanity.com/paper/2203.02833/) (2022)
- [TruncFormer: Private LLM Inference Using Only Truncations](https://mlsanity.com/paper/2412.01042/) (2024)
- [Stochastic Perturbations of Tabular Features for Non-Deterministic Inference with Automunge](https://mlsanity.com/paper/2202.09248/) (2022)
- [Securing Neural Networks with Knapsack Optimization](https://mlsanity.com/paper/2304.10442/) (2023)
- [Towards Fast and Scalable Private Inference](https://mlsanity.com/paper/2307.04077/) (2023)
- [Bi-CryptoNets: Leveraging Different-Level Privacy for Encrypted Inference](https://mlsanity.com/paper/2402.01296/) (2024)
- [Secure Multiparty Generative AI](https://mlsanity.com/paper/2409.19120/) (2024)
- [DeepShare: Sharing ReLU Across Channels and Layers for Efficient Private Inference](https://mlsanity.com/paper/2512.17398/) (2025)
- [LRD-MPC: Efficient MPC Inference through Low-rank Decomposition](https://mlsanity.com/paper/2602.14397/) (2026)

## Abstract (arXiv)

The simultaneous rise of machine learning as a service and concerns over user privacy have increasingly motivated the need for private inference (PI). While recent work demonstrates PI is possible using cryptographic primitives, the computational overheads render it impractical. The community is largely unprepared to address these overheads, as the source of slowdown in PI stems from the ReLU operator whereas optimizations for plaintext inference focus on optimizing FLOPs. In this paper we re-think the ReLU computation and propose optimizations for PI tailored to properties of neural networks. Specifically, we reformulate ReLU as an approximate sign test and introduce a novel truncation method for the sign test that significantly reduces the cost per ReLU. These optimizations result in a specific type of stochastic ReLU. The key observation is that the stochastic fault behavior is well suited for the fault-tolerant properties of neural network inference. Thus, we provide significant savings without impacting accuracy. We collectively call the optimizations Circa and demonstrate improvements of up to 4.7x storage and 3x runtime over baseline implementations; we further show that Circa can be used on top of recent PI optimizations to obtain 1.8x additional speedup.