mlsanityData to 7 Oct 2026

Personalised Distillation: Empowering Open-Sourced LLMs with Adaptive Learning for Code Generation

arXiv 2310.18628 ↗First posted 28 Oct 2023cs.CL cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

With the rise of powerful closed-sourced LLMs (ChatGPT, GPT-4), there are increasing interests in distilling the capabilies of close-sourced LLMs to smaller open-sourced LLMs. Previous distillation methods usually prompt ChatGPT to generate a set of instructions and answers, for the student model to learn. However, such standard distillation approach neglects the merits and conditions of the student model. Inspired by modern teaching principles, we design a personalised distillation process, in which the student attempts to solve a task first, then the teacher provides an adaptive refinement for the student to improve. Instead of feeding the student with teacher's prior, personalised distillation enables personalised learning for the student model, as it only learns on examples it makes mistakes upon and learns to improve its own solution. On code generation, personalised distillation consistently outperforms standard distillation with only one third of the data. With only 2.5-3K personalised examples that incur a data-collection cost of 4-6, we boost CodeGen-mono-16B by 7% to achieve 36.4% pass@1 and StarCoder by 12.2% to achieve 45.8% pass@1 on HumanEval.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 13 of its 27 references that are in the map. First posted on arXiv 28 Oct 2023.

Builds on

13 of 27 references.

References · 13

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  5. 2021Program Synthesis with Large Language Modelscited by 3,509
  6. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  8. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  9. 2023Teaching Large Language Models to Self-Debugcited by 963
  10. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  11. 2022Generating Sequences by Learning to Self-Correctcited by 271
  12. 2023Improving Code Generation by Training with Natural Language Feedbackcited by 75
1 more
  1. 2023Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimismcited by 69

Built on by

Cited by year

Citing papers per year: 2024: 7.

7 papers.

  1. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  2. 2024Generating CAD Code with Vision-Language Models for 3D Designscited by 61
  3. 2024Exploring Data-Efficient Adaptation of Large Language Models for Code Generationcited by 3
  4. 2024Error-driven Data-efficient Large Multimodal Model Tuningcited by 2
  5. 2024A Knowledge-Component-Based Methodology for Evaluating AI Assistantscited by 1
  6. 2024AMR-Evol: Adaptive Modular Response Evolution Elicits Better Knowledge Distillation for Large Language Models in Code Generationcited by 1
  7. 2024Investigating the Transferability of Code Repair for Low-Resource Programming Languagescited by 0