# Privacy-Preserving Text Classification on BERT Embeddings with Homomorphic Encryption

arXiv: https://arxiv.org/abs/2210.02574 · HTML: https://mlsanity.com/paper/2210.02574/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 26 references that are in the map.
- First posted on arXiv 5 Oct 2022.

## Builds on (most cited first, 13 of 13)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [Information Leakage in Embedding Models](https://mlsanity.com/paper/2004.00053/) (2020)
- [A Stack-Propagation Framework with Token-Level Intent Detection for Spoken Language Understanding](https://mlsanity.com/paper/1909.02188/) (2019)
- [Towards Robust and Privacy-preserving Text Representations](https://mlsanity.com/paper/1805.06093/) (2018)
- [Natural Language Understanding with Privacy-Preserving BERT](https://mlsanity.com/paper/2104.07504/) (2021)
- [Privacy-preserving Neural Representations of Text](https://mlsanity.com/paper/1808.09408/) (2018)
- [PrivFT: Private and Fast Text Classification with Homomorphic Encryption](https://mlsanity.com/paper/1908.06972/) (2019)

## Built on by (most cited first, 9 of 9)

- [Converting Transformers to Polynomial Form for Secure Inference Over Homomorphic Encryption](https://mlsanity.com/paper/2311.08610/) (2023)
- [Encryption-Friendly LLM Architecture](https://mlsanity.com/paper/2410.02486/) (2024)
- [HETAL: Efficient Privacy-preserving Transfer Learning with Homomorphic Encryption](https://mlsanity.com/paper/2403.14111/) (2024)
- [FedTherapist: Mental Health Monitoring with User-Generated Linguistic Expressions on Smartphones via Federated Learning](https://mlsanity.com/paper/2310.16538/) (2023)
- [SentinelLMs: Encrypted Input Adaptation and Fine-tuning of Language Models for Private and Secure Inference](https://mlsanity.com/paper/2312.17342/) (2023)
- [Training Large Scale Polynomial CNNs for E2E Inference over Homomorphic Encryption](https://mlsanity.com/paper/2304.14836/) (2023)
- [CRaSh: Clustering, Removing, and Sharing Enhance Fine-tuning without Full Large Language Model](https://mlsanity.com/paper/2310.15477/) (2023)
- [Enhancing Privacy in the Early Detection of Sexual Predators Through Federated Learning and Differential Privacy](https://mlsanity.com/paper/2501.12537/) (2025)
- [Which PPML Would a User Choose? A Structured Decision Support Framework for Developers to Rank PPML Techniques Based on User Acceptance Criteria](https://mlsanity.com/paper/2411.06995/) (2024)

## Abstract (arXiv)

Embeddings, which compress information in raw text into semantics-preserving low-dimensional vectors, have been widely adopted for their efficacy. However, recent research has shown that embeddings can potentially leak private information about sensitive attributes of the text, and in some cases, can be inverted to recover the original input text. To address these growing privacy challenges, we propose a privatization mechanism for embeddings based on homomorphic encryption, to prevent potential leakage of any piece of information in the process of text classification. In particular, our method performs text classification on the encryption of embeddings from state-of-the-art models like BERT, supported by an efficient GPU implementation of CKKS encryption scheme. We show that our method offers encrypted protection of BERT embeddings, while largely preserving their utility on downstream text classification tasks.