# SentinelLMs: Encrypted Input Adaptation and Fine-tuning of Language Models for Private and Secure Inference

arXiv: https://arxiv.org/abs/2312.17342 · HTML: https://mlsanity.com/paper/2312.17342/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 27 references that are in the map.
- First posted on arXiv 28 Dec 2023.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [Differentially Private Fine-tuning of Language Models](https://mlsanity.com/paper/2110.06500/) (2021)
- [Information Leakage in Embedding Models](https://mlsanity.com/paper/2004.00053/) (2020)
- [Pretrained Language Models for Text Generation: A Survey](https://mlsanity.com/paper/2105.10311/) (2021)
- [Natural Language Understanding with Privacy-Preserving BERT](https://mlsanity.com/paper/2104.07504/) (2021)
- [Private Model Compression via Knowledge Distillation](https://mlsanity.com/paper/1811.05072/) (2018)
- [Privacy-Preserving Text Classification on BERT Embeddings with Homomorphic Encryption](https://mlsanity.com/paper/2210.02574/) (2022)

## Built on by (most cited first, 9 of 9)

- [SoK: Semantic Privacy in Large Language Models](https://mlsanity.com/paper/2506.23603/) (2025)
- [From Large AI Models to Agentic AI: A Tutorial on Future Intelligent Communications](https://mlsanity.com/paper/2505.22311/) (2025)
- [AlienLM: Alienization of Language for API-Boundary Privacy in Black-Box LLMs](https://mlsanity.com/paper/2601.22710/) (2026)
- [Towards Privacy-Preserving Large Language Model: Text-free Inference Through Alignment and Adaptation](https://mlsanity.com/paper/2604.06831/) (2026)
- [Selective Token-Level Cryptographic Redaction for Privacy-Preserving Clinical Deployment of Large Language Models](https://mlsanity.com/paper/2606.03399/) (2026)
- [Privacy from Symmetry: Orthogonally Equivariant Transformers for LLM Inference](https://mlsanity.com/paper/2606.16461/) (2026)
- [DeepInvert: Semi-Supervised Embedding Inversion Against Obfuscated Language Models](https://mlsanity.com/paper/2608.04477/) (2026)
- [PrivBench: A Holistic and Modular Benchmarking Platform for Evaluating Text-to-Text Privatization](https://mlsanity.com/paper/2608.29624/) (2026)
- [A Novel Semantic Manifold Alignment Attack against Embedding-to-Embedding Obfuscation in Privacy-Preserving LLMs](https://mlsanity.com/paper/2609.06749/) (2026)

## Abstract (arXiv)

This paper addresses the privacy and security concerns associated with deep neural language models, which serve as crucial components in various modern AI-based applications. These models are often used after being pre-trained and fine-tuned for specific tasks, with deployment on servers accessed through the internet. However, this introduces two fundamental risks: (a) the transmission of user inputs to the server via the network gives rise to interception vulnerabilities, and (b) privacy concerns emerge as organizations that deploy such models store user data with restricted context. To address this, we propose a novel method to adapt and fine-tune transformer-based language models on passkey-encrypted user-specific text. The original pre-trained language model first undergoes a quick adaptation (without any further pre-training) with a series of irreversible transformations applied to the tokenizer and token embeddings. This enables the model to perform inference on encrypted inputs while preventing reverse engineering of text from model parameters and intermediate outputs. After adaptation, models are fine-tuned on encrypted versions of existing training datasets. Experimental evaluation employing adapted versions of renowned models (e.g., BERT, RoBERTa) across established benchmark English and multilingual datasets for text classification and sequence labeling shows that encrypted models achieve performance parity with their original counterparts. This serves to safeguard performance, privacy, and security cohesively.