Retrospective Distillation Attribution via Normalized Response Similarity
Not on the map. Why?
Abstract (arXiv)
Model distillation transfers capabilities through supervised fine-tuning (SFT) on teacher responses, often collected from commercial APIs, raising questions of model provenance. Existing distillation attribution methods have been largely evaluated on students immediately after the SFT step. However, a distilled model may undergo further SFT, preference optimization, or reinforcement learning before release, while an auditor may lack access to the pre-distillation checkpoint required by reference-based attribution. To close this gap, we propose SCOUT, an output-only method that aggregates recurring *syntactic patterns* into candidate profiles, filters low-contrast patterns, and calibrates student--candidate distances against inter-candidate distances. SCOUT supports attribution and abstention using only current texts, without model weights, token likelihoods, or historical checkpoints. Auditing publicly released descendants of distilled models spanning diverse post-training objectives, SCOUT consistently identifies the distillation source. Furthermore, tracing teacher-associated *syntactic signatures* along training trajectories reveals that they emerge during distillation and persist through subsequent preference optimization and reinforcement learning.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 45 of its 71 references that are in the map. First posted on arXiv 26 Sep 2026.
Builds on
45 of 71 references.
References · 45
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2025Qwen3 Technical Reportcited by 8,195
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
- 2024Qwen2.5 Technical Reportcited by 4,713
- 2025s1: Simple test-time scalingcited by 1,311
- 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
- 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
- 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
- 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
33 more
- 2023A Watermark for Large Language Modelscited by 667
- 2025Dream 7B: Diffusion Large Language Modelscited by 515
- 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Datacited by 194
- 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Modelscited by 187
- 2018PRADA: Protecting against DNN Model Stealing Attackscited by 159
- 2021On the Importance of Difficulty Calibration in Membership Inference Attackscited by 137
- 2025SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learningcited by 89
- 2025AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learningcited by 85
- 2024Instructional Fingerprinting of Large Language Modelscited by 64
- 2025dParallel: Learnable Parallel Decoding for dLLMscited by 59
- 2024Watermarking Makes Language Models Radioactivecited by 52
- 2025Exploring the Limit of Outcome Reward for Learning Mathematical Reasoningcited by 51
- 2024Detection and Measurement of Syntactic Templates in Generated Textcited by 44
- 2026d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillationcited by 34
- 2025Antidistillation Samplingcited by 24
- 2024PhyloLM : Inferring the Phylogeny of Large Language Models and Predicting their Performances in Benchmarkscited by 18
- 2025DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimizationcited by 16
- 2025DRPO: Efficient Reasoning via Decoupled Reward Policy Optimizationcited by 16
- 2025Model Provenance Testing for Large Language Modelscited by 15
- 2026Chaining the Evidence: Robust Reinforcement Learning for Deep Search Agents with Citation-Aware Rubric Rewardscited by 14
- 2025Disrupting Model Merging: A Parameter-Level Defense Without Sacrificing Accuracycited by 11
- 2025Who Taught You That? Tracing Teachers in Model Distillationcited by 8
- 2025Thinking Preference Optimizationcited by 8
- 2025Fingerprinting LLMs via Prompt Injectioncited by 8
- 2023Matching Pairs: Attributing Fine-Tuned Models to their Pre-Trained Large Language Modelscited by 6
- 2025Model Unmerging: Making Your Models Unmergeable for Secure Model Sharingcited by 6
- 2026Protecting Language Models Against Unauthorized Distillation through Trace Rewritingcited by 6
- 2026R2-dLLM: Accelerating Diffusion Large Language Models via Spatio-Temporal Redundancy Reductioncited by 6
- 2026TextSeal: A Localized LLM Watermark for Provenance & Distillation Protectioncited by 2
- 2026Self-Augmenting Retrieval for Diffusion Language Modelscited by 2
- 2026Reference-Based Distillation Detection in LLMscited by 2
- 2026Making Models Unmergeable via Scaling-Sensitive Loss Landscapecited by 1
- 2026Asking Back: Interaction-Layer Antidistillation Watermarkscited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Distill Not Only Data but Also Rewards: Can Smaller Language Models Surpass Larger Ones?
- 2026Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why
- 2026When Teachers Mislead: Spurious-Signal-Aware On-Policy Distillation
- 2026When Teacher Guidance Misleads: Reward-Aligned On-Policy Distillation
- 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes
- 2026Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models
- 2026Inductive Feedback for Mixed-Policy Distillation
- 2026A Survey of On-Policy Distillation for Large Language Models