mlsanityData to 7 Oct 2026

Code Less, Align More: Efficient LLM Fine-tuning for Code Generation with Data Pruning

arXiv 2407.05040 ↗First posted 6 Jul 2024cs.SE cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

Recent work targeting large language models (LLMs) for code generation demonstrated that increasing the amount of training data through synthetic code generation often leads to exceptional performance. In this paper we explore data pruning methods aimed at enhancing the efficiency of model training specifically for code LLMs. We present techniques that integrate various clustering and pruning metrics to selectively reduce training data without compromising the accuracy and functionality of the generated code. We observe significant redundancies in synthetic training data generation, where our experiments demonstrate that benchmark performance can be largely preserved by training on only 10% of the data. Moreover, we observe consistent improvements in benchmark results through moderate pruning of the training data. Our experiments show that these pruning strategies not only reduce the computational resources needed but also enhance the overall quality code generation.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 39 of its 51 references that are in the map. First posted on arXiv 6 Jul 2024.

Builds on

39 of 51 references.

References · 39

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2021Program Synthesis with Large Language Modelscited by 3,509
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  11. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  12. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
27 more
  1. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  2. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  3. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  4. 2023Instruction Tuning with GPT-4cited by 676
  5. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  6. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  7. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  8. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
  9. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 354
  10. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 301
  11. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 242
  12. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 201
  13. 2023Active Prompting with Chain-of-Thought for Large Language Modelscited by 154
  14. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 140
  15. 2024Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuningcited by 134
  16. 2023RTLFixer: Automatically Fixing RTL Syntax Errors with Large Language Modelscited by 98
  17. 2021Trivial or impossible -- dichotomous data difficulty masks model differences (on ImageNet and beyond)cited by 35
  18. 2023Rethinking the Instruction Quality: LIFT is What You Needcited by 33
  19. 2023Too Large; Data Reduction for Vision-Language Pre-Trainingcited by 26
  20. 2023Data Selection for Fine-tuning Large Language Models Using Transferred Shapley Valuescited by 21
  21. 2024Text-centric Alignment for Multi-Modality Learningcited by 12
  22. 2022Toward More Generalized Malicious URL Detection Modelscited by 9
  23. 2023Differential Good Arm Identificationcited by 8
  24. 2022Fast online inference for nonlinear contextual bandit based on Generative Adversarial Networkcited by 7
  25. 2024A Study in Dataset Pruning for Image Super-Resolutioncited by 6
  26. 2024On the Limitations of Embedding Based Methods for Measuring Functional Correctness for Code Generationcited by 6
  27. 2016HDBSCAN: Density based Clustering over Location Based Servicescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 14; 2026: 2.

19 papers.

Sort:

By area:

  1. 2024A Survey on Data Synthesis and Augmentation for Large Language Modelscited by 40
  2. 2025Type-Constrained Code Generation with Language Modelscited by 32
  3. 2025A Coreset Selection of Coreset Selection Literature: Introduction and Recent Advancescited by 28
  4. 2025Large Language Models for Code Generation: A Comprehensive Survey of Challenges, Techniques, Evaluation, and Applicationscited by 26
  5. 2025Every FLOP Counts: Scaling a 300B Mixture-of-Experts LING LLM without Premium GPUscited by 22
  6. 2024Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspectivecited by 15
  7. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  8. 2025Beyond Similarity: A Gradient-based Graph Method for Instruction Tuning Data Selectioncited by 7
  9. 2025SPACE: Noise Contrastive Estimation Stabilizes Self-Play Fine-Tuning for Large Language Modelscited by 7
  10. 2025Uncertainty Profiles for LLMs: Uncertainty Source Decomposition and Adaptive Model-Metric Selectioncited by 4
  11. 2025MapCoder-Lite: Distilling Multi-Agent Coding into a Single Small LLMcited by 3
  12. 2024Benchmarking Large Language Model Uncertainty for Prompt Optimizationcited by 2
7 more
  1. 2025LLM-Aided Customizable Profiling of Code Data Based On Programming Language Conceptscited by 2
  2. 2025Multimodal Chip Physical Design Engineer Assistantcited by 1
  3. 2026CODEBLOCK: Learning to Supervise Code at the Right Granularitycited by 1
  4. 2025ReCatcher: Towards LLMs Regression Testing for Code Generationcited by 0
  5. 2025Do LLMs Align with My Task? Evaluating Text-to-SQL via Dataset Alignmentcited by 0
  6. 2025Reasoning Distillation and Structural Alignment for Improved Code Generationcited by 0
  7. 2026Reward-Free Code Alignment from Pretrained or Fine-Tuned LLM: Unpacking the Trade-offs for Code Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Data-efficient LLM Fine-tuning for Code Generation
  2. 2024Improving the Ability of Pre-trained Language Model by Imparting Large Language Model's Experience ↗
  3. 2024CodeLutra: Boosting LLM Code Generation via Preference-Guided Refinement
  4. 2023Better Language Models of Code through Self-Improvement
  5. 2025Synthetic Data Generation Using Large Language Models: Advances in Text and Code
  6. 2025Alignment with Fill-In-the-Middle for Enhancing Code Generation
  7. 2025Think, Prune, Train, Improve: Scaling Reasoning without Scaling Models
  8. 2025Every Sample Matters: Leveraging Mixture-of-Experts and High-Quality Data for Efficient and Accurate Code LLM