# Meeting in the Middle: A Co-Design Paradigm for FHE and AI Inference

arXiv: https://arxiv.org/abs/2603.20504 · HTML: https://mlsanity.com/paper/2603.20504/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 25 references that are in the map.
- First posted on arXiv 20 Mar 2026.

## Builds on (most cited first, 12 of 12)

- [A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions](https://mlsanity.com/paper/2006.02903/) (2020)
- [Gazelle: A Low Latency Framework for Secure Neural Network Inference](https://mlsanity.com/paper/1801.05507/) (2018)
- [Slalom: Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware](https://mlsanity.com/paper/1806.03287/) (2018)
- [Low Latency Privacy Preserving Inference](https://mlsanity.com/paper/1812.10659/) (2018)
- [CryptoDL: Deep Neural Networks over Encrypted Data](https://mlsanity.com/paper/1711.05189/) (2017)
- [DeepReDuce: ReLU Reduction for Fast Private Inference](https://mlsanity.com/paper/2103.01396/) (2021)
- [Private and Reliable Neural Network Inference](https://mlsanity.com/paper/2210.15614/) (2022)
- [Toward Scalable Fully Homomorphic Encryption Through Light Trusted Computing Assistance](https://mlsanity.com/paper/1905.07766/) (2019)
- [Dash: Accelerating Distributed Private Convolutional Neural Network Inference with Arithmetic Garbled Circuits](https://mlsanity.com/paper/2302.06361/) (2023)
- [CAT: A GPU-Accelerated FHE Framework with Its Application to High-Precision Private Dataset Query](https://mlsanity.com/paper/2503.22227/) (2025)
- [HHEML: Hybrid Homomorphic Encryption for Privacy-Preserving Machine Learning on Edge](https://mlsanity.com/paper/2510.20243/) (2025)
- [CGF-Softmax: A Cumulant-Based Softmax Reformulation for Efficient Inference under Homomorphic Encryption](https://mlsanity.com/paper/2602.01621/) (2026)

## Abstract (arXiv)

Modern cloud inference creates a two sided privacy problem where users reveal sensitive inputs to providers, while providers must execute proprietary model weights inside potentially leaky execution environments. Fully homomorphic encryption (FHE) offers cryptographic guarantees but remains prohibitively expensive for modern architectures. We argue that progress requires co-design where specializing FHE schemes/compilers for the static structure of inference circuits, while simultaneously constraining inference architectures to reduce dominant homomorphic cost drivers. We outline a meet in the middle agenda and concrete optimization targets on both axes.