mlsanityData to 7 Oct 2026

Leave It to the Experts: Detecting Knowledge Distillation via MoE Expert Signatures

arXiv 2510.16968 ↗First posted 19 Oct 2025cs.LG cs.AI cs.CLcited by 0

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Knowledge Distillation (KD) accelerates training of large language models (LLMs) but poses intellectual property protection and LLM diversity risks. Existing KD detection methods based on self-identity or output similarity can be easily evaded through prompt engineering. We present a KD detection framework effective in both white-box and black-box settings by exploiting an overlooked signal: the transfer of MoE "structural habits", especially internal routing patterns. Our approach analyzes how different experts specialize and collaborate across various inputs, creating distinctive fingerprints that persist through the distillation process. To extend beyond the white-box setup and MoE architectures, we further propose Shadow-MoE, a black-box method that constructs proxy MoE representations via auxiliary distillation to compare these patterns between arbitrary model pairs. We establish a comprehensive, reproducible benchmark that offers diverse distilled checkpoints and an extensible framework to facilitate future research. Extensive experiments demonstrate >94% detection accuracy across various scenarios and strong robustness to prompt-based evasion, outperforming existing baselines while highlighting the structural habits transfer in LLMs.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 37 of its 47 references that are in the map. First posted on arXiv 19 Oct 2025.

Builds on

37 of 47 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Qwen3 Technical Report14 May
  2. 2024QuantMoE-Bench: Examining Post-Training Quantization for Mixture-of-Experts12 Jun
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 34

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  5. 2020Extracting Training Data from Large Language Modelscited by 2,144
  6. 2020Knowledge Distillation: A Surveycited by 1,578
  7. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  8. 2025s1: Simple test-time scalingcited by 1,311
  9. 2021Measuring Coding Challenge Competence With APPScited by 879
  10. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  11. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  12. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
22 more
  1. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  2. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  3. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  4. 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 312
  5. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
  6. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  7. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparisoncited by 249
  8. 2025Llama-Nemotron: Efficient Reasoning Modelscited by 151
  9. 2023TACO: Topics in Algorithmic COde generation datasetcited by 150
  10. 2019Thieves on Sesame Street! Model Extraction of BERT-based APIscited by 136
  11. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 129
  12. 2021Dataset Inference: Ownership Resolution in Machine Learningcited by 115
  13. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 104
  14. 2022Less is More: Task-aware Layer-wise Distillation for Language Model Compressioncited by 91
  15. 2021RiddleSense: Reasoning about Riddle Questions Featuring Linguistic Creativity and Commonsense Knowledgecited by 60
  16. 2025PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Modelscited by 57
  17. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 55
  18. 2021Undistillable: Making A Nasty Teacher That CANNOT teach studentscited by 36
  19. 2025DOGe: Defensive Output Generation for LLM Protection Against Knowledge Distillationcited by 14
  20. 2025Quantification of Large Language Model Distillationcited by 9
  21. 2025Advancing MoE Efficiency: A Collaboration-Constrained Routing (C2R) Strategy for Better Expert Parallelism Designcited by 6
  22. 2025Occult: Optimizing Collaborative Communication across Experts for Accelerated Parallel MoE Training and Inferencecited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Every Expert Matters: Towards Effective Knowledge Distillation for Mixture-of-Experts Language Models
  2. 2026KDFP: A first-principles approach to knowledge distillation in large language models
  3. 2025Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions
  4. 2025Exploiting the Experts: Unauthorized Compression in MoE-LLMs
  5. 2024Multi-Stage Balanced Distillation: Addressing Long-Tail Challenges in Sequence-Level Knowledge Distillation
  6. 2024Efficient Knowledge Injection in LLMs via Self-Distillation
  7. 2025A Dual-Space Framework for General Knowledge Distillation of Large Language Models
  8. 2026WASD: Wasserstein-based Knowledge Distillation for Large Language Models