# Exploiting Unstructured Sparsity in Fully Homomorphic Encrypted DNNs

arXiv: https://arxiv.org/abs/2503.09184 · HTML: https://mlsanity.com/paper/2503.09184/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 41 references that are in the map.
- First posted on arXiv 12 Mar 2025.

## Builds on (most cited first, 15 of 15)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures](https://mlsanity.com/paper/1607.03250/) (2016)
- [Sparse GPU Kernels for Deep Learning](https://mlsanity.com/paper/2006.10901/) (2020)
- [TenSEAL: A Library for Encrypted Tensor Operations Using Homomorphic Encryption](https://mlsanity.com/paper/2104.03152/) (2021)
- [TRADI: Tracking deep neural network weight distributions for uncertainty estimation](https://mlsanity.com/paper/1912.11316/) (2019)
- [When Homomorphic Encryption Marries Secret Sharing: Secure Large-Scale Sparse Logistic Regression and Applications in Risk Control](https://mlsanity.com/paper/2008.08753/) (2020)
- [Precise Approximation of Convolutional Neural Networks for Homomorphically Encrypted Data](https://mlsanity.com/paper/2105.10879/) (2021)
- [The Efficiency Spectrum of Large Language Models: An Algorithmic Survey](https://mlsanity.com/paper/2312.00678/) (2023)
- [Exploiting Data Sparsity in Secure Cross-Platform Social Recommendation](https://mlsanity.com/paper/2202.07253/) (2022)
- [Designing Efficient LLM Accelerators for Edge Devices](https://mlsanity.com/paper/2408.00462/) (2024)
- [SECDA: Efficient Hardware/Software Co-Design of FPGA-based DNN Accelerators for Edge Inference](https://mlsanity.com/paper/2110.00478/) (2021)
- [ML with HE: Privacy Preserving Machine Learning Inferences for Genome Studies](https://mlsanity.com/paper/2110.11446/) (2021)
- [EDLaaS: Fully Homomorphic Encryption Over Neural Network Graphs for Vision and Private Strawberry Yield Forecasting](https://mlsanity.com/paper/2110.13638/) (2021)

## Built on by (most cited first, 2 of 2)

- [Efficient Privacy-Preserving Recommendation on Sparse Data using Fully Homomorphic Encryption](https://mlsanity.com/paper/2509.03024/) (2025)
- [GPU Acceleration of Sparse Fully Homomorphic Encrypted DNNs](https://mlsanity.com/paper/2604.11659/) (2026)

## Abstract (arXiv)

The deployment of deep neural networks (DNNs) in privacy-sensitive environments is constrained by computational overheads in fully homomorphic encryption (FHE). This paper explores unstructured sparsity in FHE matrix multiplication schemes as a means of reducing this burden while maintaining model accuracy requirements. We demonstrate that sparsity can be exploited in arbitrary matrix multiplication, providing runtime benefits compared to a baseline naive algorithm at all sparsity levels. This is a notable departure from the plaintext domain, where there is a trade-off between sparsity and the overhead of the sparse multiplication algorithm. In addition, we propose three sparse multiplication schemes in FHE based on common plaintext sparse encodings. We demonstrate the performance gain is scheme-invariant; however, some sparse schemes vastly reduce the memory storage requirements of the encrypted matrix at high sparsity values. Our proposed sparse schemes yield an average performance gain of 2.5x at 50% unstructured sparsity, with our multi-threading scheme providing a 32.5x performance increase over the equivalent single-threaded sparse computation when utilizing 64 cores.