# Identified-Set Geometry of Distributional Model Extraction under Top-K Censored API Access

arXiv: https://arxiv.org/abs/2605.10407 · HTML: https://mlsanity.com/paper/2605.10407/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 12 of its 14 references that are in the map.
- First posted on arXiv 11 May 2026.

## Placed in

- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 12 of 12)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [Stealing Machine Learning Models via Prediction APIs](https://mlsanity.com/paper/1609.02943/) (2016)
- [High Accuracy and High Fidelity Extraction of Neural Networks](https://mlsanity.com/paper/1909.01838/) (2019)
- [Stealing Part of a Production Language Model](https://mlsanity.com/paper/2403.06634/) (2024)
- [MiniLLM: On-Policy Distillation of Large Language Models](https://mlsanity.com/paper/2306.08543/) (2023)
- [Truncation Sampling as Language Model Desmoothing](https://mlsanity.com/paper/2210.15191/) (2022)
- [Logits of API-Protected LLMs Leak Proprietary Information](https://mlsanity.com/paper/2403.09539/) (2024)
- [Sequences of Logits Reveal the Low Rank Structure of Language Models](https://mlsanity.com/paper/2510.24966/) (2025)
- [Provably Learning from Modern Language Models via Low Logit Rank](https://mlsanity.com/paper/2512.09892/) (2025)

## Built on by (most cited first, 1 of 1)

- [When Top-K Misses the Decision: Tool-Call Drift in Multi-Teacher On-Policy Distillation](https://mlsanity.com/paper/2607.07050/) (2026)

## Abstract (arXiv)

Modern LLM APIs often reveal only top-K logit scores and censor the remaining vocabulary. We study the per-position distribution-recovery limits of this access model. For censoring threshold τ, the compatible teacher distributions form an identified set whose total-variation diameter is exactly UK=(V-K)exp(τ)/(ZA+(V-K)exp(τ)), where ZA is the observed partition function. For KL recovery, we give a computable binary-endpoint lower bound and an asymptotically matching small-ambiguity upper bound, with an extension to reference-aware attackers. Experiments on a Qwen3 math-reasoning teacher reveal a layered extraction hierarchy: on-task top-K distillation recovers 12% of private capability, full-logit distillation recovers 56% despite 99% KL closure, and generation-based extraction recovers 96%. Top-K censoring therefore limits per-position distribution recovery but does not by itself prevent capability extraction, separating fidelity from transfer in prompt-only logit distillation.