# EmojiPrompt: Generative Prompt Obfuscation for Privacy-Preserving Communication with Cloud-based LLMs

arXiv: https://arxiv.org/abs/2402.05868 · HTML: https://mlsanity.com/paper/2402.05868/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 59 references that are in the map.
- First posted on arXiv 8 Feb 2024.

## Builds on (most cited first, 22 of 22)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [Scalable Extraction of Training Data from (Production) Language Models](https://mlsanity.com/paper/2311.17035/) (2023)
- [Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)](https://mlsanity.com/paper/2203.13366/) (2022)
- [Split learning for health: Distributed deep learning without sharing raw patient data](https://mlsanity.com/paper/1812.00564/) (2018)
- [Distributed learning of deep neural network over multiple agents](https://mlsanity.com/paper/1810.06060/) (2018)
- [Gazelle: A Low Latency Framework for Secure Neural Network Inference](https://mlsanity.com/paper/1801.05507/) (2018)
- [Is ChatGPT a Good Recommender? A Preliminary Study](https://mlsanity.com/paper/2304.10149/) (2023)
- [Full Parameter Fine-tuning for Large Language Models with Limited Resources](https://mlsanity.com/paper/2306.09782/) (2023)
- [Data-efficient Fine-tuning for LLM-based Recommendation](https://mlsanity.com/paper/2401.17197/) (2024)
- [How to Index Item IDs for Recommendation Foundation Models](https://mlsanity.com/paper/2305.06569/) (2023)
- [CrypTFlow2: Practical 2-Party Secure Inference](https://mlsanity.com/paper/2010.06457/) (2020)
- [Differential Privacy for Text Analytics via Natural Text Sanitization](https://mlsanity.com/paper/2106.01221/) (2021)
- [ReLLa: Retrieval-enhanced Large Language Models for Lifelong Sequential Behavior Comprehension in Recommendation](https://mlsanity.com/paper/2308.11131/) (2023)
- [DP-Forward: Fine-tuning and Inference on Language Models with Differential Privacy in Forward Pass](https://mlsanity.com/paper/2309.06746/) (2023)
- [GenRec: Large Language Model for Generative Recommendation](https://mlsanity.com/paper/2307.00457/) (2023)
- [Natural Language Understanding with Privacy-Preserving BERT](https://mlsanity.com/paper/2104.07504/) (2021)
- [Hide and Seek (HaS): A Lightweight Framework for Prompt Privacy Protection](https://mlsanity.com/paper/2309.03057/) (2023)
- [Towards Differentially Private Text Representations](https://mlsanity.com/paper/2006.14170/) (2020)
- [Protecting User Privacy in Remote Conversational Systems: A Privacy-Preserving framework based on text sanitization](https://mlsanity.com/paper/2306.08223/) (2023)
- [LLMs Can Understand Encrypted Prompt: Towards Privacy-Computing Friendly Transformers](https://mlsanity.com/paper/2305.18396/) (2023)

## Built on by (most cited first, 9 of 9)

- [MoralBench: Moral Evaluation of LLMs](https://mlsanity.com/paper/2406.04428/) (2024)
- [PRISM: Privacy-Aware Routing for Adaptive Cloud-Edge LLM Inference via Semantic Sketch Collaboration](https://mlsanity.com/paper/2511.22788/) (2025)
- [AlienLM: Alienization of Language for API-Boundary Privacy in Black-Box LLMs](https://mlsanity.com/paper/2601.22710/) (2026)
- [Towards Privacy-Preserving LLM Inference via Covariant Obfuscation (Technical Report)](https://mlsanity.com/paper/2603.01499/) (2026)
- [Your Inference Request Will Become a Black Box: Confidential Inference for Cloud-based Large Language Models](https://mlsanity.com/paper/2603.00196/) (2026)
- [SplitAgent: A Privacy-Preserving Distributed Architecture for Enterprise-Cloud Agent Collaboration](https://mlsanity.com/paper/2603.08221/) (2026)
- [PragLocker: Protecting Agent Intellectual Property in Untrusted Deployments via Non-Portable Prompts](https://mlsanity.com/paper/2605.05974/) (2026)
- [Selective Token-Level Cryptographic Redaction for Privacy-Preserving Clinical Deployment of Large Language Models](https://mlsanity.com/paper/2606.03399/) (2026)
- [P2Skill: Privacy Preserving Skill Distillation for Cloud-Local LLM Inference Systems](https://mlsanity.com/paper/2608.14094/) (2026)

## Abstract (arXiv)

Cloud-based Large Language Models (LLMs) such as ChatGPT have become increasingly integral to daily operations. Nevertheless, they also introduce privacy concerns: firstly, numerous studies underscore the risks to user privacy posed by jailbreaking cloud-based LLMs; secondly, the LLM service providers have access to all user data, which deters individuals from confidently utilizing such services. To address such concerns, we propose a simple yet effective paradigm, EmojiPrompt, to protect user privacy. At its core, EmojiPrompt performs generative transformation, obfuscating private data within prompts with linguistic and non-linguistic elements before submitting them to cloud-based LLMs. We evaluate EmojiPrompt's performance across 8 datasets from various domains. We also propose simulated inference attacks to assess EmojiPrompt's ability to preserve user privacy. The results demonstrate that EmojiPrompt effectively obfuscates user private data, while largely maintaining, or even enhancing, performances compared to the unobfuscated version. Furthermore, EmojiPrompt's atomic-level obfuscation allows it to function exclusively with cloud-based LLMs. For source code, please refer to: https://github.com/agiresearch/EmojiCrypt.