Identified-Set Geometry of Distributional Model Extraction under Top-K Censored API Access
On the map: Knowledge distillation → On-policy and self-distillation for LLMs
Abstract (arXiv)
Modern LLM APIs often reveal only top-K logit scores and censor the remaining vocabulary. We study the per-position distribution-recovery limits of this access model. For censoring threshold τ, the compatible teacher distributions form an identified set whose total-variation diameter is exactly UK=(V-K)exp(τ)/(ZA+(V-K)exp(τ)), where ZA is the observed partition function. For KL recovery, we give a computable binary-endpoint lower bound and an asymptotically matching small-ambiguity upper bound, with an extension to reference-aware attackers. Experiments on a Qwen3 math-reasoning teacher reveal a layered extraction hierarchy: on-task top-K distillation recovers 12% of private capability, full-logit distillation recovers 56% despite 99% KL closure, and generation-based extraction recovers 96%. Top-K censoring therefore limits per-position distribution recovery but does not by itself prevent capability extraction, separating fidelity from transfer in prompt-only logit distillation.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 12 of its 14 references that are in the map. First posted on arXiv 11 May 2026.
Builds on
12 of 14 references.
References · 12
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
- 2021Program Synthesis with Large Language Modelscited by 3,637
- 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
- 2016Stealing Machine Learning Models via Prediction APIscited by 619
- 2019High Accuracy and High Fidelity Extraction of Neural Networkscited by 214
- 2024Stealing Part of a Production Language Modelcited by 159
- 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 139
- 2022Truncation Sampling as Language Model Desmoothingcited by 123
- 2024Logits of API-Protected LLMs Leak Proprietary Informationcited by 40
- 2025Sequences of Logits Reveal the Low Rank Structure of Language Modelscited by 9
- 2025Provably Learning from Modern Language Models via Low Logit Rankcited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Beyond Indistinguishability: Measuring Extraction Risk in LLM APIs
- 2026A Token-Level Analysis of Sampled-Token Reverse-KL On-Policy Distillation
- 2026DistillGuard: Evaluating Defenses Against LLM Knowledge Distillation
- 2026The Gradient That Top-k Distillation Never Computes
- 2026Leaky Students: Membership Inference against On-Policy Distillation
- 2026Bounded Behavioral Indistinguishability for Black-Box LLM Distillation
- 2024Theoretical guarantees on the best-of-n alignment policy
- 2026Mismatch Matters: On-Policy Distillation Beyond Token Agreement