# FEnc2: Unifying Data Packing for Efficient Private Inference via Convolution and Architecture-Aware Fragment Encoding

arXiv: https://arxiv.org/abs/2606.16359 · HTML: https://mlsanity.com/paper/2606.16359/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 70 references that are in the map.
- First posted on arXiv 15 Jun 2026.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [Gazelle: A Low Latency Framework for Secure Neural Network Inference](https://mlsanity.com/paper/1801.05507/) (2018)
- [Low Latency Privacy Preserving Inference](https://mlsanity.com/paper/1812.10659/) (2018)
- [TenSEAL: A Library for Encrypted Tensor Operations Using Homomorphic Encryption](https://mlsanity.com/paper/2104.03152/) (2021)
- [LinGCN: Structural Linearized Graph Convolutional Network for Homomorphically Encrypted Inference](https://mlsanity.com/paper/2309.14331/) (2023)
- [HEMET: A Homomorphic-Encryption-Friendly Privacy-Preserving Mobile Neural Network Architecture](https://mlsanity.com/paper/2106.00038/) (2021)
- [F1: A Fast and Programmable Accelerator for Fully Homomorphic Encryption (Extended Version)](https://mlsanity.com/paper/2109.05371/) (2021)
- [HeLayers: A Tile Tensors Framework for Large Neural Networks on Encrypted Data](https://mlsanity.com/paper/2011.01805/) (2020)
- [EVA: An Encrypted Vector Arithmetic Language and Compiler for Efficient Homomorphic Computation](https://mlsanity.com/paper/1912.11951/) (2019)
- [Orion: A Fully Homomorphic Encryption Framework for Deep Learning](https://mlsanity.com/paper/2311.03470/) (2023)
- [BTS: An Accelerator for Bootstrappable Fully Homomorphic Encryption](https://mlsanity.com/paper/2112.15479/) (2021)
- [ARK: Fully Homomorphic Encryption Accelerator with Runtime Data Generation and Inter-Operation Key Reuse](https://mlsanity.com/paper/2205.00922/) (2022)
- [Secure Human Action Recognition by Encrypted Neural Network Inference](https://mlsanity.com/paper/2104.09164/) (2021)
- [CryptoGCN: Fast and Scalable Homomorphically Encrypted Graph Convolutional Network Inference](https://mlsanity.com/paper/2209.11904/) (2022)
- [TensorFHE: Achieving Practical Computation on Encrypted Data Using GPGPU](https://mlsanity.com/paper/2212.14191/) (2022)
- [NeuJeans: Private Neural Network Inference with Joint Optimization of Convolution and FHE Bootstrapping](https://mlsanity.com/paper/2312.04356/) (2023)
- [AEGIS: Scaling Long-Sequence Homomorphic Encrypted Transformer Inference via Hybrid Parallelism on Multi-GPU Systems](https://mlsanity.com/paper/2604.03425/) (2026)
- [slytHErin: An Agile Framework for Encrypted Deep Neural Network Inference](https://mlsanity.com/paper/2305.00690/) (2023)
- [Toward Lossless Homomorphic Encryption for Scientific Computation](https://mlsanity.com/paper/2309.07284/) (2023)
- [EFFACT: A Highly Efficient Full-Stack FHE Acceleration Platform](https://mlsanity.com/paper/2504.15817/) (2025)

## Abstract (arXiv)

Fully Homomorphic Encryption (FHE) enables privacy-preserving machine learning but incurs extreme computational and memory overhead. These costs come not only from expensive low-level primitives, including Number Theoretic Transform (NTT), rotation, and key-switching, but also from inefficient ciphertext packing at the application level. Existing packing strategies typically preserve either neighboring data elements or feature grouping, but not both, leading to wasted ciphertext slots, excessive rotations, and inflated ciphertext counts. We propose FEnc2, a unified and principled fragment-based encoding framework for CKKS-based private convolutional neural network inference. FEnc2 optimizes slot utilization, rotation complexity, and ciphertext density through two components: 1)Conv-aware Encoding, which analytically selects an optimal fragment size to decouple spatial dependencies and jointly minimize inner-outer rotations across layers, and 2)Arch-aware Ct Compression, which restores ciphertext density after feature- or channel-reduction layers. Together, these transformations reshape encrypted workload structure and reduce homomorphic operations by one to two orders of magnitude. With full memory capacity utilized, i.e., at maximum batch size, FEnc2 achieves end-to-end latency speedups over the state-of-the-art Orion of up to 228.83x on GPU and 226.06x on CPU for LeNet on MNIST, and up to 4.55x on GPU and 9.43x on CPU for MobileNet on ImageNet. FEnc2 is hardware-agnostic yet architecturally transformative: by optimizing encrypted tensor layout before execution, it reduces ciphertext count and workload pressure on hardware, complementing primitive-level optimizations such as NTT and keyswitch accelerators. These results show that application-level data layout is a first-order architectural design dimension for encrypted inference and an important enabler for next-generation FHE systems.