mlsanityData to 14 Oct 2026

Stealing Part of a Production Language Model

arXiv 2403.06634 ↗First posted 11 Mar 2024cs.CRcited by 159

Not on the map. Why?

Cited by 35 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce the first model-stealing attack that extracts precise, nontrivial information from black-box production language models like OpenAI's ChatGPT or Google's PaLM-2. Specifically, our attack recovers the embedding projection layer (up to symmetries) of a transformer model, given typical API access. For under 20 USD, our attack extracts the entire projection matrix of OpenAI's Ada and Babbage language models. We thereby confirm, for the first time, that these black-box models have a hidden dimension of 1024 and 2048, respectively. We also recover the exact hidden dimension size of the gpt-3.5-turbo model, and estimate it would cost under 2,000 in queries to recover the entire projection matrix. We conclude with potential defenses and mitigations, and discuss the implications of possible future work that could extend our attack.

Key facts. Cited by 159 papers in the map (data through 14 Oct 2026). First posted on arXiv 11 Mar 2024.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 41; 2025: 62; 2026: 56.

159 papers.

Sort:

By area:

  1. 2024Does Prompt Formatting Have Any Impact on LLM Performance?cited by 141
  2. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 137
  3. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  4. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  5. 2024Machine Against the RAG: Jamming Retrieval-Augmented Generation with Blocker Documentscited by 102
  6. 2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 96
  7. 2025Prompt Flow Integrity to Prevent Privilege Escalation in LLM Agentscited by 87
  8. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 65
  9. 2024LLMmap: Fingerprinting For Large Language Modelscited by 58
  10. 2025Harnessing the Universal Geometry of Embeddingscited by 55
  11. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 46
  12. 2024Extracting Prompts by Inverting LLM Outputscited by 41
38 more
  1. 2024Logits of API-Protected LLMs Leak Proprietary Informationcited by 40
  2. 2025Are You Getting What You Pay For? Auditing Model Substitution in LLM APIscited by 38
  3. 2025Red Teaming the Mind of the Machine: A Systematic Evaluation of Prompt Injection and Jailbreak Vulnerabilities in LLMscited by 34
  4. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 33
  5. 2024MiniPLM: Knowledge Distillation for Pre-Training Language Modelscited by 29
  6. 2024What Happened in LLMs Layers when Trained for Fast vs. Slow Thinking: A Gradient Perspectivecited by 29
  7. 2025We Should Chart an Atlas of All the World's Modelscited by 25
  8. 2025Antidistillation Samplingcited by 25
  9. 2024Black-Box Detection of Language Model Watermarkscited by 23
  10. 2025Privacy Auditing of Large Language Modelscited by 23
  11. 2024ConfusedPilot: Confused Deputy Risks in RAG-based LLMscited by 22
  12. 2025Frontier AI's Impact on the Cybersecurity Landscapecited by 21
  13. 2024Generative AI Misuse: A Taxonomy of Tactics and Insights from Real-World Datacited by 20
  14. 2024GASLITEing the Retrieval: Exploring Vulnerabilities in Dense Embedding-based Searchcited by 20
  15. 2025Auditing Prompt Caching in Language Model APIscited by 20
  16. 2025Reality Check: A New Evaluation Ecosystem Is Necessary to Understand AI's Real World Effectscited by 19
  17. 2025A Survey on Model Extraction Attacks and Defenses for Large Language Modelscited by 19
  18. 2024Hide and Seek: Fingerprinting Large Language Models with Evolutionary Learningcited by 18
  19. 2024Encryption-Friendly LLM Architecturecited by 17
  20. 2024Deep Linear Probe Generators for Weight Space Learningcited by 17
  21. 2025Fun-tuning: Characterizing the Vulnerability of Proprietary LLMs to Optimization-based Prompt Injection Attacks via the Fine-Tuning Interfacecited by 17
  22. 2024A Watermark for Black-Box Language Modelscited by 16
  23. 2025GaussMarker: Robust Dual-Domain Watermark for Diffusion Modelscited by 16
  24. 2024Just rephrase it! Uncertainty estimation in closed-source language models via multiple rephrased queriescited by 15
  25. 2024LLM Whisperer: An Inconspicuous Attack to Bias LLM Responsescited by 15
  26. 2025Model Provenance Testing for Large Language Modelscited by 15
  27. 2025SoK: Large Language Model Copyright Auditing via Fingerprintingcited by 15
  28. 2024SLIP: Securing LLMs IP Using Weights Decompositioncited by 14
  29. 2025Scalable Fingerprinting of Large Language Modelscited by 14
  30. 2024Evaluating and Mitigating IP Infringement in Visual Generative AIcited by 13
  31. 2024Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data?cited by 13
  32. 2024LeakAgent: RL-based Red-teaming Agent for LLM Privacy Leakagecited by 13
  33. 2025Unveiling the Basin-Like Loss Landscape in Large Language Modelscited by 13
  34. 2025Is Your LLM Overcharging You? Tokenization, Transparency, and Incentivescited by 13
  35. 2024Open LLMs are Necessary for Current Private Adaptations and Outperform their Closed Alternativescited by 12
  36. 2026A Systematic Survey of Security Threats and Defenses in LLM-Based AI Agents: A Layered Attack Surface Frameworkcited by 11
  37. 2025Invisible Tokens, Visible Bills: The Urgent Need to Audit Hidden Operations in Opaque LLM Servicescited by 10
  38. 2025Log Probability Tracking of LLM APIscited by 10

50 most cited of 159 All 159 (JSON)