# FastQuery: Communication-efficient Embedding Table Query for Private LLM Inference

arXiv: https://arxiv.org/abs/2405.16241 · HTML: https://mlsanity.com/paper/2405.16241/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 3 references that are in the map.
- First posted on arXiv 25 May 2024.

## Builds on (most cited first, 2 of 2)

- [ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers](https://mlsanity.com/paper/2206.01861/) (2022)
- [Characterizing and Optimizing End-to-End Systems for Private Inference](https://mlsanity.com/paper/2207.07177/) (2022)

## Built on by (most cited first, 1 of 1)

- [EfficientNav: Towards On-Device Object-Goal Navigation with Navigation Map Caching and Retrieval](https://mlsanity.com/paper/2510.18546/) (2025)

## Abstract (arXiv)

With the fast evolution of large language models (LLMs), privacy concerns with user queries arise as they may contain sensitive information. Private inference based on homomorphic encryption (HE) has been proposed to protect user query privacy. However, a private embedding table query has to be formulated as a HE-based matrix-vector multiplication problem and suffers from enormous computation and communication overhead. We observe the overhead mainly comes from the neglect of 1) the one-hot nature of user queries and 2) the robustness of the embedding table to low bit-width quantization noise. Hence, in this paper, we propose a private embedding table query optimization framework, dubbed FastQuery. FastQuery features a communication-aware embedding table quantization algorithm and a one-hot-aware dense packing algorithm to simultaneously reduce both the computation and communication costs. Compared to prior-art HE-based frameworks, e.g., Cheetah, Iron, and Bumblebee, FastQuery achieves more than 4.3×, 2.7×, 1.3× latency reduction, respectively and more than 75.7×, 60.2×, 20.2× communication reduction, respectively, on both LLAMA-7B and LLAMA-30B.