mlsanityData to 14 Oct 2026

Asking Back: Interaction-Layer Antidistillation Watermarks

arXiv 2605.16462 ↗First posted 15 May 2026cs.CR cs.AIcited by 2

On the map: Knowledge distillation → Federated, robust and private distillation

Abstract (arXiv)

Detecting unauthorized knowledge distillation from a deployed LLM API is hard because the defender controls neither the attacker's training pipeline nor the next-token logits. Existing defenses operate on the teacher's output tokens -- biasing the next-token distribution (green-list watermarks, cryptographic schemes, antidistillation sampling) or rewriting outputs after generation. Recent work shows a paraphrasing attacker can strip these signals without losing the underlying knowledge. We propose interaction-layer antidistillation watermarks, which move the trace one layer higher, into the teacher's interaction behavior: the defender wraps the teacher with a system prompt that intermittently induces a behavioral marker -- an explicit follow-up question, a low-frequency variant, or a declarative restatement. An oblivious distiller inherits the behavior, and the defender audits via black-box queries with a human-validated LLM-as-judge (Cohen's kappa = 0.84/0.78 on strong/style rubrics). Across 63 LoRA-distilled students under a Llama-3.3-70B-Instruct teacher (35,343 judged samples), behavioral watermarks transfer at 88.9% (Gemma) / 80.9% (OLMo) / 45.2% (Qwen) relative fidelity (H1, H2). Under non-adaptive DIPPER paraphrasing, robustness decomposes into a teacher-self ceiling (about 66.4%) and student-relative retention of 21-112%, with OLMo preserving the watermark above the teacher itself (H3, F-Amp). Low-density (about 20%) explicit and implicit declarative variants transfer above per-family baseline (H4, F-Style). An N=20 in-lab study (pre-registered Latin-square) shows all marker variants within 0.22 Likert step of baseline; TOST, Friedman, and Bonferroni-Wilcoxon support H5. The interaction layer is a viable design locus for antidistillation watermarking, complementary to token-, model-, and reasoning-trace-layer defenses.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation. Builds on 34 of its 42 references that are in the map. First posted on arXiv 15 May 2026.

Builds on

34 of 42 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026Protecting Language Models Against Unauthorized Distillation through Trace Rewriting16 Feb
  2. 2025DOGe: Defensive Output Generation for LLM Protection Against Knowledge Distillation26 May
  3. 2025Can LLM Watermarks Robustly Prevent Unauthorized Knowledge Distillation?17 Feb

Other references · 31

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2021Program Synthesis with Large Language Modelscited by 3,637
  7. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  8. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  9. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  10. 2025Gemma 3 Technical Reportcited by 1,824
  11. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  12. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
19 more
  1. 2023A Watermark for Large Language Modelscited by 725
  2. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  3. 2019Neural Text Generation with Unlikelihood Trainingcited by 503
  4. 2023Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defensecited by 429
  5. 2023Can AI-Generated Text be Reliably Detected?cited by 386
  6. 20242 OLMo 2 Furiouscited by 340
  7. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 256
  8. 2023Undetectable Watermarks for Language Modelscited by 226
  9. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Datacited by 200
  10. 2024Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questionscited by 76
  11. 2024Instructional Fingerprinting of Large Language Modelscited by 68
  12. 2020Radioactive data: tracing through trainingcited by 67
  13. 2024Watermarking Makes Language Models Radioactivecited by 58
  14. 2025Idiosyncrasies in Large Language Modelscited by 33
  15. 2025Antidistillation Samplingcited by 25
  16. 2025Who Taught You That? Tracing Teachers in Model Distillationcited by 8
  17. 2025Information-Preserving Reformulation of Reasoning Traces for Antidistillationcited by 8
  18. 2025DITTO: A Spoofing Attack Framework on Watermarked LLMs via Knowledge Distillationcited by 6
  19. 2026Antidistillation Fingerprintingcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Retrospective Distillation Attribution via Normalized Response Similaritycited by 0
  2. 2026Radioactive Feedback: Tracing Models Trained by Label-Only LLM Supervisioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Unified attacks to large language model watermarks: spoofing and scrubbing in unauthorized knowledge distillation
  2. 2026SkillWatermark: An Embedded Skill Watermark of Progressive Privacy Inference via Benign Prompts ↗
  3. 2026DistillGuard: Evaluating Defenses Against LLM Knowledge Distillation
  4. 2026Adaptively Robust LLM Monitoring via Activation Watermarking
  5. 2024Watermark Stealing in Large Language Models
  6. 2026Language Model Fingerprinting Requires Rethinking Watermark Teachers
  7. 2026Learning to Follow In-Context Watermark Instructions via Self-Distillation
  8. 2024Discovering Spoofing Attempts on Language Model Watermarks