# Power-Softmax: Towards Secure LLM Inference over Encrypted Data

arXiv: https://arxiv.org/abs/2410.09457 · HTML: https://mlsanity.com/paper/2410.09457/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 48 references that are in the map.
- First posted on arXiv 12 Oct 2024.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023)
- [Analyzing the Structure of Attention in a Transformer Language Model](https://mlsanity.com/paper/1906.04284/) (2019)
- [On Protecting the Data Privacy of Large Language Models (LLMs): A Survey](https://mlsanity.com/paper/2403.05156/) (2024)
- [THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption](https://mlsanity.com/paper/2206.00216/) (2022)
- [PUMA: Secure Inference of LLaMA-7B in Five Minutes](https://mlsanity.com/paper/2307.12533/) (2023)
- [Pi-nets: Deep Polynomial Neural Networks](https://mlsanity.com/paper/2003.03828/) (2020)
- [Precise Approximation of Convolutional Neural Networks for Homomorphically Encrypted Data](https://mlsanity.com/paper/2105.10879/) (2021)
- [HeLayers: A Tile Tensors Framework for Large Neural Networks on Encrypted Data](https://mlsanity.com/paper/2011.01805/) (2020)
- [LLMs Can Understand Encrypted Prompt: Towards Privacy-Computing Friendly Transformers](https://mlsanity.com/paper/2305.18396/) (2023)
- [Primer: Fast Private Transformer Inference on Encrypted Data](https://mlsanity.com/paper/2303.13679/) (2023)
- [Converting Transformers to Polynomial Form for Secure Inference Over Homomorphic Encryption](https://mlsanity.com/paper/2311.08610/) (2023)
- [AutoFHE: Automated Adaption of CNNs for Efficient Evaluation over FHE](https://mlsanity.com/paper/2310.08012/) (2023)
- [A methodology for training homomorphicencryption friendly neural networks](https://mlsanity.com/paper/2111.03362/) (2021)
- [MERGE: Fast Private Text Generation](https://mlsanity.com/paper/2305.15769/) (2023)
- [East: Efficient and Accurate Secure Transformer Framework for Inference](https://mlsanity.com/paper/2308.09923/) (2023)
- [Efficient Skip Connections Realization for Secure Inference on Encrypted Data](https://mlsanity.com/paper/2306.06736/) (2023)
- [Neural Networks with (Low-Precision) Polynomial Approximations: New Insights and Techniques for Accuracy Improvement](https://mlsanity.com/paper/2402.11224/) (2024)

## Built on by (most cited first, 7 of 7)

- [Private Transformer Inference in MLaaS: A Survey](https://mlsanity.com/paper/2505.10315/) (2025)
- [Key-Conditioned Orthonormal Transform Gating (K-OTG): Multi-Key Access Control with Hidden-State Scrambling for LoRA-Tuned Models](https://mlsanity.com/paper/2512.17519/) (2025)
- [CGF-Softmax: A Cumulant-Based Softmax Reformulation for Efficient Inference under Homomorphic Encryption](https://mlsanity.com/paper/2602.01621/) (2026)
- [Towards Privacy-Preserving Machine Translation at the Inference Stage: A New Task and Benchmark](https://mlsanity.com/paper/2603.14756/) (2026)
- [HEAT: Faster Fully Homomorphic Inference via Approximations-Weights Co-Adaptation](https://mlsanity.com/paper/2609.01730/) (2026)
- [DegreeSpar: Structured Degree Sparsity for Efficient Secure Transformer Inference](https://mlsanity.com/paper/2609.32204/) (2026)
- [A Graph-Based Analysis of Comparability, Scale, and Attribution in Private Transformer Inference](https://mlsanity.com/paper/2610.19607/) (2026)

## Abstract (arXiv)

Modern cryptographic methods for implementing privacy-preserving LLMs such as glsHE require the LLMs to have a polynomial form. Forming such a representation is challenging because transformers include non-polynomial components, such as Softmax and layer normalization. Previous approaches have either directly approximated pre-trained models with large-degree polynomials, which are less efficient over HE, or replaced non-polynomial components with easier-to-approximate primitives before training, e.g., Softmax with pointwise attention. The latter approach might introduce scalability challenges. We present a new HE-friendly variant of self-attention that offers a stable form for training and is easy to approximate with polynomials for secure inference. Our work introduces the first polynomial LLMs over a billion parameters, exceeding the size of previous models by more than tenfold. The resulting models demonstrate reasoning and in-context learning (ICL) capabilities comparable to standard transformers of the same size, representing a breakthrough in the field. Finally, we provide a detailed latency breakdown for each computation over encrypted data, paving the way for further optimization, and explore the differences in inductive bias between models relying on our HE-friendly variant and standard transformers.