mlsanityData to 7 Oct 2026

Logits of API-Protected LLMs Leak Proprietary Information

arXiv 2403.09539 ↗First posted 14 Mar 2024cs.CL cs.AI cs.CRcited by 38

Not on the map. Why?

Abstract (arXiv)

Large language model (LLM) providers often hide the architectural details and parameters of their proprietary models by restricting public access to a limited API. In this work we show that, with only a conservative assumption about the model architecture, it is possible to learn a surprisingly large amount of non-public information about an API-protected LLM from a relatively small number of API queries (e.g., costing under 1000 USD for OpenAI's gpt-3.5-turbo). Our findings are centered on one key observation: most modern LLMs suffer from a softmax bottleneck, which restricts the model outputs to a linear subspace of the full output space. We exploit this fact to unlock several capabilities, including (but not limited to) obtaining cheap full-vocabulary outputs, auditing for specific types of model updates, identifying the source LLM given a single full LLM output, and even efficiently discovering the LLM's hidden size. Our empirical investigations show the effectiveness of our methods, which allow us to estimate the embedding size of OpenAI's gpt-3.5-turbo to be about 4096. Lastly, we discuss ways that LLM providers can guard against these attacks, as well as how these capabilities can be viewed as a feature (rather than a bug) by allowing for greater transparency and accountability.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 24 of its 27 references that are in the map. First posted on arXiv 14 Mar 2024.

Builds on

24 of 27 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  6. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  7. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  8. 2016Stealing Machine Learning Models via Prediction APIscited by 602
  9. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  10. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  11. 2015Inverting Visual Representations with Convolutional Networkscited by 168
  12. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 164
12 more
  1. 2024Stealing Part of a Production Language Modelcited by 152
  2. 2017Breaking the Softmax Bottleneck: A High-Rank RNN Language Modelcited by 147
  3. 2019Thieves on Sesame Street! Model Extraction of BERT-based APIscited by 136
  4. 2023Auditing large language models: a three-layered approachcited by 135
  5. 2023Language Model Inversioncited by 66
  6. 2023CodeFusion: A Pre-trained Diffusion Model for Code Generationcited by 31
  7. 2023Closing the Curious Case of Neural Text Degenerationcited by 30
  8. 2021Models In a Spelling Bee: Language Models Implicitly Learn the Character Composition of Tokenscited by 26
  9. 2020Stolen Probability: A Structural Weakness of Neural Language Modelscited by 21
  10. 2022Low-Rank Softmax Can Have Unargmaxable Classes in Theory but Rarely in Practicecited by 14
  11. 2023Towards Publicly Accountable Frontier LLMs: Building an External Scrutiny Ecosystem under the ASPIRE Frameworkcited by 11
  12. 2019Distilling BERT into Simple Neural Networks with Unlabeled Transfer Datacited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 17; 2026: 9.

38 papers.

Sort:

By area:

  1. 2024Large Language Models for Education: A Survey and Outlookcited by 185
  2. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  3. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  4. 2024Aligning LLMs with Individual Preferences via Interactioncited by 63
  5. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  6. 2024Extracting Prompts by Inverting LLM Outputscited by 40
  7. 2025Are You Getting What You Pay For? Auditing Model Substitution in LLM APIscited by 35
  8. 2025Auditing Prompt Caching in Language Model APIscited by 19
  9. 2025SimMark: A Robust Sentence-Level Similarity-Based Watermarking Algorithm for Large Language Modelscited by 13
  10. 2025Scalable Fingerprinting of Large Language Modelscited by 11
  11. 2025Predicting the Performance of Black-box LLMs through Follow-up Queriescited by 9
  12. 2025Mining Intrinsic Rewards from LLM Hidden States for Efficient Best-of-N Samplingcited by 9
26 more
  1. 2025Better Language Model Inversion by Compactly Representing Next-Token Distributionscited by 9
  2. 2024A Middle Path for On-Premises LLM Deployment: Preserving Privacy Without Sacrificing Model Confidentialitycited by 8
  3. 2025Sequences of Logits Reveal the Low Rank Structure of Language Modelscited by 8
  4. 2024Zer0-Jack: A Memory-efficient Gradient-based Jailbreaking Method for Black-box Multi-modal Large Language Modelscited by 6
  5. 2024UniOQA: A Unified Framework for Knowledge Graph Question Answering with Large Language Modelscited by 5
  6. 2026Channel Location Constrains the Auditability of Subliminal Learningcited by 5
  7. 2025Approximating Language Model Training Data from Weightscited by 4
  8. 2024A Survey on Privacy Attacks Against Digital Twin Systems in AI-Roboticscited by 3
  9. 2025Every Language Model Has a Forgery-Resistant Signaturecited by 3
  10. 2024An Efficient Matrix Multiplication Algorithm for Accelerating Inference in Binary and Ternary Neural Networkscited by 2
  11. 2025LLM Hypnosis: Exploiting User Feedback for Unauthorized Knowledge Injection to All Userscited by 2
  12. 2026Disentangling Geometry, Performance, and Training in Language Modelscited by 2
  13. 2024Preserving Knowledge in Large Language Model with Model-Agnostic Self-Decompressioncited by 1
  14. 2025Beyond Release: Access Considerations for Generative AI Systemscited by 1
  15. 2025Distribution Prompting: Understanding the Expressivity of Language Models Through the Next-Token Distributions They Can Producecited by 1
  16. 2025A Rose by Any Other Name Would Smell as Sweet: Categorical Homotopy Theory for Large Language Modelscited by 1
  17. 2025Provably Extracting the Features from a General Superpositioncited by 1
  18. 2026Identified-Set Geometry of Distributional Model Extraction under Top-K Censored API Accesscited by 1
  19. 2024Pre-trained Encoder Inference: Revealing Upstream Encoders In Downstream Machine Learning Servicescited by 0
  20. 2025FiMMIA: scaling semantic perturbation-based membership inference across modalitiescited by 0
  21. 2026SCoOP: Semantic Consistent Opinion Pooling for Uncertainty Quantification in Multiple Vision-Language Model Systemscited by 0
  22. 2026The Geometry of Last-Layer Model Stealingcited by 0
  23. 2026Black-Box Inference of LLM Architectural Properties with Restrictive API Accesscited by 0
  24. 2026Jagged Judges: Epistemic Stability Under Perturbation, Pressure, and Persistencecited by 0
  25. 2026Auditing and Mitigating Privacy Leakage in Cloud-Edge Collaborative Decodingcited by 0
  26. 2026CUSP: Decomposable Collective Uncertainty for Multi-Agent Multimodal Reasoningcited by 0