# Standing on the Shoulders of Giants: Stabilized Knowledge Distillation for Cross--Language Code Clone Detection

arXiv: https://arxiv.org/abs/2605.02860 · HTML: https://mlsanity.com/paper/2605.02860/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 30 of its 53 references that are in the map.
- First posted on arXiv 4 May 2026.

## Placed in

- Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/)

## Builds on (most cited first, 30 of 30)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention](https://mlsanity.com/paper/2303.16199/) (2023)
- [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020)
- [Textbooks Are All You Need](https://mlsanity.com/paper/2306.11644/) (2023)
- [UniXcoder: Unified Cross-Modal Pre-training for Code Representation](https://mlsanity.com/paper/2203.03850/) (2022)
- [Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters](https://mlsanity.com/paper/2212.10001/) (2022)
- [CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks](https://mlsanity.com/paper/2105.12655/) (2021)
- [Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thought](https://mlsanity.com/paper/2501.04682/) (2025)
- [Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency](https://mlsanity.com/paper/2309.17272/) (2023)
- [Phi-3 Safety Post-Training: Aligning Language Models with a "Break-Fix" Cycle](https://mlsanity.com/paper/2407.13833/) (2024)
- [Beyond Accuracy: Evaluating Self-Consistency of Code Large Language Models with IdentityChain](https://mlsanity.com/paper/2310.14053/) (2023)
- [Compressing Pre-trained Models of Code into 3 MB](https://mlsanity.com/paper/2208.07120/) (2022)
- [Greening Large Language Models of Code](https://mlsanity.com/paper/2309.04076/) (2023)
- [Towards Understanding the Capability of Large Language Models on Code Clone Detection: A Survey](https://mlsanity.com/paper/2308.01191/) (2023)
- [LLM2: Let Large Language Models Harness System 2 Reasoning](https://mlsanity.com/paper/2412.20372/) (2024)
- [Generalizability of Code Clone Detection on CodeBERT](https://mlsanity.com/paper/2208.12588/) (2022)
- [CCT-Code: Cross-Consistency Training for Multilingual Clone Detection and Code Search](https://mlsanity.com/paper/2305.11626/) (2023)
- [Scaling Laws Behind Code Understanding Model](https://mlsanity.com/paper/2402.12813/) (2024)
- [Towards Better Code Understanding in Decoder-Only Models with Contrastive Learning](https://mlsanity.com/paper/2406.12326/) (2024)
- [On the Compression of Language Models for Code: An Empirical Study on CodeBERT](https://mlsanity.com/paper/2412.13737/) (2024)
- [Cross-Language Source Code Clone Detection Using Deep Learning with InferCode](https://mlsanity.com/paper/2205.04913/) (2022)
- [An Empirical Study of Knowledge Distillation for Code Understanding Tasks](https://mlsanity.com/paper/2508.15423/) (2025)

## Abstract (arXiv)

Cross-language code clone detection (X-CCD) is challenging because semantically equivalent programs written in different languages often share little surface similarity. Although large language models (LLMs) have shown promise for semantic clone detection, their use as black-box systems raises concerns about cost, reproducibility, privacy, and unreliable output formatting. In particular, compact open-source models often struggle to follow reasoning-oriented prompts and to produce outputs that can be consistently mapped to binary clone labels. To address these limitations, we propose a knowledge distillation framework that transfers reasoning capabilities from DeepSeek-R1 into compact open-source student models for X-CCD. Using cross-language code pairs derived from Project CodeNet, we construct reasoning-oriented synthetic training data and fine-tune Phi3 and Qwen-Coder with LoRA adapters. We further introduce response stabilization methods, including forced conclusion prompting, a binary classification head, and a contrastive classification head, and evaluate model behavior using both predictive metrics and response rate. Experiments on Python--Java, Rust--Java, Rust--Python, and Rust--Ruby show that knowledge distillation consistently improves the reliability of compact models and often improves predictive performance, especially under distribution shift. In addition, classification-head variants substantially reduce inference time compared to generation-based inference. Overall, our results show that reasoning-oriented distillation combined with response stabilization makes compact open-source models more practical and reliable for X-CCD detection.