# Cachemir: Fully Homomorphic Encrypted Inference of Generative Large Language Model with KV Cache

arXiv: https://arxiv.org/abs/2602.11470 · HTML: https://mlsanity.com/paper/2602.11470/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Feb 2026.

## Built on by (most cited first, 5 of 5)

- [Public-Decay Homomorphic State Space Models for Private Sequence Inference](https://mlsanity.com/paper/2605.16647/) (2026)
- [HEAT: Faster Fully Homomorphic Inference via Approximations-Weights Co-Adaptation](https://mlsanity.com/paper/2609.01730/) (2026)
- [HE-Guardrail: A Homomorphic Guardrail Against Jailbreak Attacks for Encrypted Large Language Model Inference](https://mlsanity.com/paper/2609.21484/) (2026)
- [Static Bootstrap Placement for Encrypted Language Model Decoding](https://mlsanity.com/paper/2610.04912/) (2026)
- [A Graph-Based Analysis of Comparability, Scale, and Attribution in Private Transformer Inference](https://mlsanity.com/paper/2610.19607/) (2026)

## Abstract (arXiv)

Generative large language models (LLMs) have revolutionized multiple domains. Modern LLMs predominantly rely on an autoregressive decoding strategy, which generates output tokens sequentially and employs a key-value cache (KV cache) to avoid redundant computation. However, the widespread deployment of LLMs has raised serious privacy concerns, as users are feeding all types of data into the model, motivating the development of secure inference frameworks based on fully homomorphic encryption (FHE). A major limitation of existing FHE-based frameworks is their inability to effectively integrate the KV cache, resulting in prohibitively high latency for autoregressive decoding. In this paper, we propose Cachemir, a KV Cache Accelerated Homomorphic Encrypted LLM Inference Regime to overcome this limitation. Cachemir comprises three key technical contributions: 1) a set of novel HE packing algorithms specifically designed to leverage the computational advantages of the KV cache; 2) an interleaved replicated packing algorithm to efficiently compute the vector-matrix multiplications that result from using the KV cache in Transformer linear layers; and 3) an augmented bootstrapping placement strategy that accounts for the KV cache to minimize bootstrapping cost. We demonstrate that Cachemir achieves 48.83× and 67.16× speedup over MOAI (ICML'25) and THOR (CCS'25) respectively on CPU and consumes less than 100 seconds on GPU to generate an output token for Llama-3-8B.