AKD : Adversarial Knowledge Distillation For Large Language Models Alignment on Coding tasks
On the map: Knowledge distillation → Knowledge distillation for LLMs
Abstract (arXiv)
The widespread adoption of Large Language Models (LLMs) for code generation, exemplified by GitHub CopilotfootnoteA coding extension powered by a Code-LLM to assist in code completion tasks surpassing a million users, highlights the transformative potential of these tools in improving developer productivity. However, this rapid growth also underscores critical concerns regarding the quality, safety, and reliability of the code they generate. As Code-LLMs evolve, they face significant challenges, including the diminishing returns of model scaling and the scarcity of new, high-quality training data. To address these issues, this paper introduces Adversarial Knowledge Distillation (AKD), a novel approach that leverages adversarially generated synthetic datasets to distill the capabilities of larger models into smaller, more efficient ones. By systematically stress-testing and refining the reasoning capabilities of Code-LLMs, AKD provides a framework for enhancing model robustness, reliability, and security while improving their parameter-efficiency. We believe this work represents a critical step toward ensuring dependable automated code generation within the constraints of existing data and the cost-efficiency of model execution.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 18 of its 29 references that are in the map. First posted on arXiv 5 May 2025.
Builds on
18 of 29 references.
References · 18
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
- 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
- 2021Program Synthesis with Large Language Modelscited by 3,637
- 2024Mixtral of Expertscited by 1,823
- 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
- 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
- 2019A Comprehensive Survey on Transfer Learningcited by 1,160
- 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,060
6 more
- 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 358
- 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 244
- 2020MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classificationcited by 187
- 2024Evaluating Language Models for Efficient Code Generationcited by 73
- 2021Asymmetric self-play for automatic goal discovery in robotic manipulationcited by 72
- 2023DeceptPrompt: Exploiting LLM-driven Code Generation via Adversarial Natural Language Instructionscited by 33
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions
- 2024A Survey on Knowledge Distillation of Large Language Models
- 2023Lion: Adversarial Distillation of Proprietary Large Language Models
- 2025Chain-of-Code Collapse: Reasoning Failures in LLMs via Adversarial Prompting in Code Generation
- 2024Large Language Models as Robust Data Generators in Software Analytics: Are We There Yet?
- 2026Estimating the Black-box LLM Uncertainty with Distribution-Aligned Adversarial Distillation
- 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application
- 2025AdversariaLLM: A Unified and Modular Toolbox for LLM Robustness Research