# HeLayers: A Tile Tensors Framework for Large Neural Networks on Encrypted Data

arXiv: https://arxiv.org/abs/2011.01805 · HTML: https://mlsanity.com/paper/2011.01805/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Nov 2020.

## Built on by (most cited first, 22 of 22)

- [FedML-HE: An Efficient Homomorphic-Encryption-Based Privacy-Preserving Federated Learning System](https://mlsanity.com/paper/2303.10837/) (2023)
- [Converting Transformers to Polynomial Form for Secure Inference Over Homomorphic Encryption](https://mlsanity.com/paper/2311.08610/) (2023)
- [A methodology for training homomorphicencryption friendly neural networks](https://mlsanity.com/paper/2111.03362/) (2021)
- [HyPHEN: A Hybrid Packing Method and Optimizations for Homomorphic Encryption-Based Neural Networks](https://mlsanity.com/paper/2302.02407/) (2023)
- [Power-Softmax: Towards Secure LLM Inference over Encrypted Data](https://mlsanity.com/paper/2410.09457/) (2024)
- [Training Large Scale Polynomial CNNs for E2E Inference over Homomorphic Encryption](https://mlsanity.com/paper/2304.14836/) (2023)
- [Efficient Decoding Methods for Language Models on Encrypted Data](https://mlsanity.com/paper/2509.08383/) (2025)
- [Efficient Pruning for Machine Learning Under Homomorphic Encryption](https://mlsanity.com/paper/2207.03384/) (2022)
- [AEGIS: Scaling Long-Sequence Homomorphic Encrypted Transformer Inference via Hybrid Parallelism on Multi-GPU Systems](https://mlsanity.com/paper/2604.03425/) (2026)
- [HE-MAN -- Homomorphically Encrypted MAchine learning with oNnx models](https://mlsanity.com/paper/2302.08260/) (2023)
- [Efficient Skip Connections Realization for Secure Inference on Encrypted Data](https://mlsanity.com/paper/2306.06736/) (2023)
- [A Framework for Double-Blind Federated Adaptation of Foundation Models](https://mlsanity.com/paper/2502.01289/) (2025)
- [DESIGN: Encrypted GNN Inference via Server-Side Input Graph Pruning](https://mlsanity.com/paper/2507.05649/) (2025)
- [mPSAuth: Privacy-Preserving and Scalable Authentication for Mobile Web Applications](https://mlsanity.com/paper/2210.04777/) (2022)
- [High-Resolution Convolutional Neural Networks on Homomorphically Encrypted Data via Sharding Ciphertexts](https://mlsanity.com/paper/2306.09189/) (2023)
- [MetisFL: An Embarrassingly Parallelized Controller for Scalable & Efficient Federated Learning Workflows](https://mlsanity.com/paper/2311.00334/) (2023)
- [FHE-Agent: Automating CKKS Configuration for Practical Encrypted Inference via an LLM-Guided Agentic Framework](https://mlsanity.com/paper/2511.18653/) (2025)
- [PADER: Paillier-based Secure Decentralized Social Recommendation](https://mlsanity.com/paper/2601.10212/) (2026)
- [FEnc2: Unifying Data Packing for Efficient Private Inference via Convolution and Architecture-Aware Fragment Encoding](https://mlsanity.com/paper/2606.16359/) (2026)
- [Detecting and Understanding Vulnerabilities in Fully Homomorphic Encryption Frameworks](https://mlsanity.com/paper/2606.22519/) (2026)
- [OptiPrime: Optimizing Private Inference through Protocol-Hardware Co-design](https://mlsanity.com/paper/2609.16898/) (2026)
- [Fiona: Accelerating FHE Inference with Packing-Aware Ternary Weights](https://mlsanity.com/paper/2609.35352/) (2026)

## Abstract (arXiv)

Privacy-preserving solutions enable companies to offload confidential data to third-party services while fulfilling their government regulations. To accomplish this, they leverage various cryptographic techniques such as Homomorphic Encryption (HE), which allows performing computation on encrypted data. Most HE schemes work in a SIMD fashion, and the data packing method can dramatically affect the running time and memory costs. Finding a packing method that leads to an optimal performant implementation is a hard task. We present a simple and intuitive framework that abstracts the packing decision for the user. We explain its underlying data structures and optimizer, and propose a novel algorithm for performing 2D convolution operations. We used this framework to implement an HE-friendly version of AlexNet, which runs in three minutes, several orders of magnitude faster than other state-of-the-art solutions that only use HE.