mlsanityData to 14 Oct 2026

AMR-Evol: Adaptive Modular Response Evolution Elicits Better Knowledge Distillation for Large Language Models in Code Generation

arXiv 2410.00558 ↗First posted 1 Oct 2024cs.CL cs.AI cs.SEcited by 1

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

The impressive performance of proprietary LLMs like GPT4 in code generation has led to a trend to replicate these capabilities in open-source models through knowledge distillation (e.g. Code Evol-Instruct). However, these efforts often neglect the crucial aspect of response quality, relying heavily on teacher models for direct response distillation. This paradigm, especially for complex instructions, can degrade the quality of synthesized data, compromising the knowledge distillation process. To this end, our study introduces the Adaptive Modular Response Evolution (AMR-Evol) framework, which employs a two-stage process to refine response distillation. The first stage, modular decomposition, breaks down the direct response into more manageable sub-modules. The second stage, adaptive response evolution, automatically evolves the response with the related function modules. Our experiments with three popular code benchmarks (HumanEval, MBPP, and EvalPlus) attest to the superiority of the AMR-Evol framework over baseline response distillation methods. By comparing with the open-source Code LLMs trained on a similar scale of data, we observed performance enhancements: more than +3.0 points on HumanEval-Plus and +1.0 points on MBPP-Plus, which underscores the effectiveness of our framework. Our codes are available at https://github.com/ChiYeungLaw/AMR-Evol.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 35 of its 53 references that are in the map. First posted on arXiv 1 Oct 2024.

Builds on

35 of 53 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Personalised Distillation: Empowering Open-Sourced LLMs with Adaptive Learning for Code Generation28 Oct

Other references · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2021Program Synthesis with Large Language Modelscited by 3,637
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  9. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  10. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  11. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  12. 2023LIMA: Less Is More for Alignmentcited by 1,100
22 more
  1. 2023Teaching Large Language Models to Self-Debugcited by 988
  2. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 906
  3. 2021Measuring Coding Challenge Competence With APPScited by 902
  4. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  5. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  6. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  7. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  8. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
  9. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  10. 2023Textbooks Are All You Needcited by 569
  11. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  12. 2022CodeT: Code Generation with Generated Testscited by 384
  13. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 333
  14. 2022UL2: Unifying Language Learning Paradigmscited by 295
  15. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 247
  16. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 192
  17. 2024CodeGemma: Open Code Models Based on Gemmacited by 150
  18. 2024Best Practices and Lessons Learned on Synthetic Datacited by 106
  19. 2024A Survey on Data Selection for LLM Instruction Tuningcited by 77
  20. 2023CodeChain: Towards Modular Code Generation Through Chain of Self-revisions with Representative Sub-modulescited by 58
  21. 2024Beyond Chain-of-Thought: A Survey of Chain-of-X Paradigms for LLMscited by 48
  22. 2023InstructCoder: Instruction Tuning Large Language Models for Code Editingcited by 29

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026TraceCoder: A Trace-Driven Multi-Agent Framework for Automated Debugging of LLM-Generated Codecited by 6

Similar, not cited

Close in wording, no citation either way

  1. 2023SelfEvolve: A Code Evolution Framework via Large Language Models
  2. 2025RefineCoder: Iterative Improving of Large Language Models via Adaptive Critique Refinement for Code Generation
  3. 2024EVOR: Evolving Retrieval for Code Generation
  4. 2024EvoWiki: Evaluating LLMs on Evolving Knowledge
  5. 2026Enhancing Automated Unit Test Generation for NLP Libraries Using Large Language Models ↗
  6. 2025DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs
  7. 2024Evolutionary Contrastive Distillation for Language Model Alignment
  8. 2025Satori-SWE: Evolutionary Test-Time Scaling for Sample-Efficient Software Engineering