mlsanityData to 14 Oct 2026

Lossless Anti-Distillation Sampling

arXiv 2605.18829 ↗First posted 12 May 2026cs.LG cs.CRcited by 1

Not on the map. Why?

Abstract (arXiv)

Frontier commercial generative models face a growing threat from distillation, whereby a distiller harvests generated responses and trains a competing model at drastically lower cost. Existing defenses either modify the generation to degrade distillation performance, sacrificing response quality, or rely on behavioral detection mechanisms that can be readily bypassed through multi-account querying. In this work, we propose Lossless Anti-Distillation Sampling (LADS), which leaves the generation itself unchanged while substantially reducing the effectiveness of distillation. Concretely, LADS controls the latent randomness underlying inference through a coupling mechanism that preserves within-account generation independence while inducing cross-account dependence. By construction, each benign user, who typically holds only a single account, receives the same experience under LADS as they would without any defense, thereby enjoying a lossless experience. However, for a multi-account task-specific distiller, semantically similar queries submitted across different accounts are assigned coupled randomness, inducing dependence in the harvested data and thereby degrading the generalization performance of the distilled model. Using uniform convergence theory, we show that LADS provably degrades the distiller's generalization gap relative to standard i.i.d. sampling. Experiments on image generation, mathematical reasoning, and code generation confirm that LADS substantially degrades the performance of distilled students while preserving exact statistical fidelity for individual users.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 54 of its 82 references that are in the map. First posted on arXiv 12 May 2026.

Builds on

54 of 82 references.

References · 54

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  10. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  11. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  12. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
38 more
  1. 2017Billion-scale similarity search with GPUscited by 2,497
  2. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  3. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  4. 2016Practical Black-Box Attacks against Machine Learningcited by 903
  5. 2016Sequence-Level Knowledge Distillationcited by 801
  6. 2019Contrastive Representation Distillationcited by 748
  7. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  8. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  9. 2016Stealing Machine Learning Models via Prediction APIscited by 619
  10. 2018Born Again Neural Networkscited by 546
  11. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 402
  12. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
  13. 2024The Faiss librarycited by 376
  14. 2018Turning Your Weakness Into a Strength: Watermarking Deep Neural Networks by Backdooringcited by 357
  15. 2015Norm-Based Capacity Control in Neural Networkscited by 299
  16. 2017Size-Independent Sample Complexity of Neural Networkscited by 290
  17. 2022Teaching Small Language Models to Reasoncited by 284
  18. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 256
  19. 2017Embedding Watermarks into Deep Neural Networkscited by 248
  20. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 222
  21. 2019High Accuracy and High Fidelity Extraction of Neural Networkscited by 214
  22. 2018PRADA: Protecting against DNN Model Stealing Attackscited by 165
  23. 2024Stealing Part of a Production Language Modelcited by 159
  24. 2014A* Samplingcited by 151
  25. 2020Entangled Watermarks as a Defense against Model Extractioncited by 149
  26. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  27. 2016A vector-contraction inequality for Rademacher complexitiescited by 142
  28. 2019Thieves on Sesame Street! Model Extraction of BERT-based APIscited by 139
  29. 2020Data-Free Model Extractioncited by 124
  30. 2024DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solvingcited by 95
  31. 2023Protecting Language Generation Models via Invisible Watermarkingcited by 90
  32. 2019Prediction Poisoning: Towards Defenses Against DNN Model Stealing Attackscited by 81
  33. 2020Imitation Attacks and Defenses for Black-box Machine Translation Systemscited by 81
  34. 2024Watermarking Makes Language Models Radioactivecited by 58
  35. 2019Defending Against Model Stealing Attacks with Adaptive Misinformationcited by 57
  36. 2023How Smooth Is Attention?cited by 44
  37. 2021Undistillable: Making A Nasty Teacher That CANNOT teach studentscited by 36
  38. 2025MathFusion: Enhancing Mathematical Problem-solving of LLM through Instruction Fusioncited by 30

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2026Reference-Based Distillation Detection in LLMs
  2. 2025One-Step Offline Distillation of Diffusion-based Models via Koopman Modeling
  3. 2021Enhancing Data-Free Adversarial Distillation with Activation Regularization and Virtual Interpolation
  4. 2026Understanding Off- vs On-Policy Distillation: A Tale of Distinct Training Objectives
  5. 2026Poster: A Preliminary Study of LLM Distillation Inference
  6. 2024Small Scale Data-Free Knowledge Distillation
  7. 2022Revisiting Self-Distillation
  8. 2023DisWOT: Student Architecture Search for Distillation WithOut Training