Lossless Anti-Distillation Sampling
Not on the map. Why?
Abstract (arXiv)
Frontier commercial generative models face a growing threat from distillation, whereby a distiller harvests generated responses and trains a competing model at drastically lower cost. Existing defenses either modify the generation to degrade distillation performance, sacrificing response quality, or rely on behavioral detection mechanisms that can be readily bypassed through multi-account querying. In this work, we propose Lossless Anti-Distillation Sampling (LADS), which leaves the generation itself unchanged while substantially reducing the effectiveness of distillation. Concretely, LADS controls the latent randomness underlying inference through a coupling mechanism that preserves within-account generation independence while inducing cross-account dependence. By construction, each benign user, who typically holds only a single account, receives the same experience under LADS as they would without any defense, thereby enjoying a lossless experience. However, for a multi-account task-specific distiller, semantically similar queries submitted across different accounts are assigned coupled randomness, inducing dependence in the harvested data and thereby degrading the generalization performance of the distilled model. Using uniform convergence theory, we show that LADS provably degrades the distiller's generalization gap relative to standard i.i.d. sampling. Experiments on image generation, mathematical reasoning, and code generation confirm that LADS substantially degrades the performance of distilled students while preserving exact statistical fidelity for individual users.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 54 of its 82 references that are in the map. First posted on arXiv 12 May 2026.
Builds on
54 of 82 references.
References · 54
- 2020Denoising Diffusion Probabilistic Modelscited by 21,076
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
- 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
- 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
38 more
- 2017Billion-scale similarity search with GPUscited by 2,497
- 2014FitNets: Hints for Thin Deep Netscited by 1,653
- 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
- 2016Practical Black-Box Attacks against Machine Learningcited by 903
- 2016Sequence-Level Knowledge Distillationcited by 801
- 2019Contrastive Representation Distillationcited by 748
- 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
- 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
- 2016Stealing Machine Learning Models via Prediction APIscited by 619
- 2018Born Again Neural Networkscited by 546
- 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 402
- 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
- 2024The Faiss librarycited by 376
- 2018Turning Your Weakness Into a Strength: Watermarking Deep Neural Networks by Backdooringcited by 357
- 2015Norm-Based Capacity Control in Neural Networkscited by 299
- 2017Size-Independent Sample Complexity of Neural Networkscited by 290
- 2022Teaching Small Language Models to Reasoncited by 284
- 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 256
- 2017Embedding Watermarks into Deep Neural Networkscited by 248
- 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 222
- 2019High Accuracy and High Fidelity Extraction of Neural Networkscited by 214
- 2018PRADA: Protecting against DNN Model Stealing Attackscited by 165
- 2024Stealing Part of a Production Language Modelcited by 159
- 2014A* Samplingcited by 151
- 2020Entangled Watermarks as a Defense against Model Extractioncited by 149
- 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
- 2016A vector-contraction inequality for Rademacher complexitiescited by 142
- 2019Thieves on Sesame Street! Model Extraction of BERT-based APIscited by 139
- 2020Data-Free Model Extractioncited by 124
- 2024DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solvingcited by 95
- 2023Protecting Language Generation Models via Invisible Watermarkingcited by 90
- 2019Prediction Poisoning: Towards Defenses Against DNN Model Stealing Attackscited by 81
- 2020Imitation Attacks and Defenses for Black-box Machine Translation Systemscited by 81
- 2024Watermarking Makes Language Models Radioactivecited by 58
- 2019Defending Against Model Stealing Attacks with Adaptive Misinformationcited by 57
- 2023How Smooth Is Attention?cited by 44
- 2021Undistillable: Making A Nasty Teacher That CANNOT teach studentscited by 36
- 2025MathFusion: Enhancing Mathematical Problem-solving of LLM through Instruction Fusioncited by 30
50 most cited of 54 All 54 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Reference-Based Distillation Detection in LLMs
- 2025One-Step Offline Distillation of Diffusion-based Models via Koopman Modeling
- 2021Enhancing Data-Free Adversarial Distillation with Activation Regularization and Virtual Interpolation
- 2026Understanding Off- vs On-Policy Distillation: A Tale of Distinct Training Objectives
- 2026Poster: A Preliminary Study of LLM Distillation Inference
- 2024Small Scale Data-Free Knowledge Distillation
- 2022Revisiting Self-Distillation
- 2023DisWOT: Student Architecture Search for Distillation WithOut Training