# What Was Your Prompt? A Remote Keylogging Attack on AI Assistants

arXiv: https://arxiv.org/abs/2403.09751 · HTML: https://mlsanity.com/paper/2403.09751/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 42 references that are in the map.
- First posted on arXiv 14 Mar 2024.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [Quantifying Memorization Across Neural Language Models](https://mlsanity.com/paper/2202.07646/) (2022)
- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [Scalable Extraction of Training Data from (Production) Language Models](https://mlsanity.com/paper/2311.17035/) (2023)
- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023)
- [Cache Telepathy: Leveraging Shared Resource Attacks to Learn DNN Architectures](https://mlsanity.com/paper/1808.04761/) (2018)
- [Stealing Neural Networks via Timing Side Channels](https://mlsanity.com/paper/1812.11720/) (2018)
- [Privacy Side Channels in Machine Learning Systems](https://mlsanity.com/paper/2309.05610/) (2023)
- [I Know What You See: Power Side-Channel Attack on Convolutional Neural Network Accelerators](https://mlsanity.com/paper/1803.05847/) (2018)
- [Security Analysis of Deep Neural Networks Operating in the Presence of Cache Side-Channel Attacks](https://mlsanity.com/paper/1810.03487/) (2018)
- [How to 0wn NAS in Your Spare Time](https://mlsanity.com/paper/2002.06776/) (2020)
- [Bicycle Attacks Considered Harmful: Quantifying the Damage of Widespread Password Length Leakage](https://mlsanity.com/paper/2002.01513/) (2020)
- [A Review and Comparison of AI Enhanced Side Channel Analysis](https://mlsanity.com/paper/2402.02299/) (2024)

## Built on by (most cited first, 26 of 26)

- [AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathways](https://mlsanity.com/paper/2406.02630/) (2024)
- [Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)](https://mlsanity.com/paper/2407.14937/) (2024)
- [Lessons From Red Teaming 100 Generative AI Products](https://mlsanity.com/paper/2501.07238/) (2025)
- [Auditing Prompt Caching in Language Model APIs](https://mlsanity.com/paper/2502.07776/) (2025)
- [Remote Timing Attacks on Efficient Language Model Inference](https://mlsanity.com/paper/2410.17175/) (2024)
- [Exposing LLM User Privacy via Traffic Fingerprint Analysis: A Study of Privacy Risks in LLM Agent Interactions](https://mlsanity.com/paper/2510.07176/) (2025)
- [A Jailbroken GenAI Model Can Cause Substantial Harm: GenAI-powered Applications are Vulnerable to PromptWares](https://mlsanity.com/paper/2408.05061/) (2024)
- [When Speculation Spills Secrets: Side Channels via Speculative Decoding In LLMs](https://mlsanity.com/paper/2411.01076/) (2024)
- [Beyond Data Privacy: New Privacy Risks for Large Language Models](https://mlsanity.com/paper/2509.14278/) (2025)
- [Network-Level Prompt and Trait Leakage in Local Research Agents](https://mlsanity.com/paper/2508.20282/) (2025)
- [Towards Action Hijacking of Large Language Model-based Agent](https://mlsanity.com/paper/2412.10807/) (2024)
- [Chain-of-Trigger: An Agentic Backdoor that Paradoxically Enhances Agentic Robustness](https://mlsanity.com/paper/2510.08238/) (2025)
- [Memory Backdoor Attacks on Neural Networks](https://mlsanity.com/paper/2411.14516/) (2024)
- [Whisper Leak: a side-channel attack on Large Language Models](https://mlsanity.com/paper/2511.03675/) (2025)
- [Spill The Beans: Exploiting CPU Cache Side-Channels to Leak Tokens from Large Language Models](https://mlsanity.com/paper/2505.00817/) (2025)
- [Securing Educational LLMs: A Generalised Taxonomy of Attacks on LLMs and DREAD Risk Assessment](https://mlsanity.com/paper/2508.08629/) (2025)
- [Rethinking Latency Denial-of-Service: Attacking the LLM Serving Framework, Not the Model](https://mlsanity.com/paper/2602.07878/) (2026)
- [LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systems](https://mlsanity.com/paper/2509.10682/) (2025)
- [OptiLeak: Efficient Prompt Reconstruction via Reinforcement Learning in Multi-tenant LLM Services](https://mlsanity.com/paper/2602.20595/) (2026)
- [The Proxy Knows Too Much: Sealing LLM API Routers with Attested TEEs](https://mlsanity.com/paper/2606.16358/) (2026)
- [A Large-Scale Benchmark and Risk Assessment of Traffic Analysis Attacks on Cloud LLM Services](https://mlsanity.com/paper/2609.31877/) (2026)
- [ASRJam: Human-Friendly AI Speech Jamming to Prevent Automated Phone Scams](https://mlsanity.com/paper/2506.11125/) (2025)
- [Conformal Prediction for Offensive Security](https://mlsanity.com/paper/2609.05165/) (2026)
- [Detokenization Leaks: Reconstructing Local LLM Outputs From Cache Traces](https://mlsanity.com/paper/2609.06674/) (2026)
- [A Novel Semantic Manifold Alignment Attack against Embedding-to-Embedding Obfuscation in Privacy-Preserving LLMs](https://mlsanity.com/paper/2609.06749/) (2026)
- [SparLeak: Privacy Leakage from Sparse Attention in LLM Inference on Shared GPUs](https://mlsanity.com/paper/2609.38830/) (2026)

## Abstract (arXiv)

AI assistants are becoming an integral part of society, used for asking advice or help in personal and confidential issues. In this paper, we unveil a novel side-channel that can be used to read encrypted responses from AI Assistants over the web: the token-length side-channel. We found that many vendors, including OpenAI and Microsoft, have this side-channel. However, inferring the content of a response from a token-length sequence alone proves challenging. This is because tokens are akin to words, and responses can be several sentences long leading to millions of grammatically correct sentences. In this paper, we show how this can be overcome by (1) utilizing the power of a large language model (LLM) to translate these sequences, (2) providing the LLM with inter-sentence context to narrow the search space and (3) performing a known-plaintext attack by fine-tuning the model on the target model's writing style. Using these methods, we were able to accurately reconstruct 29% of an AI assistant's responses and successfully infer the topic from 55% of them. To demonstrate the threat, we performed the attack on OpenAI's ChatGPT-4 and Microsoft's Copilot on both browser and API traffic.