# Code Less, Align More: Efficient LLM Fine-tuning for Code Generation with Data Pruning

arXiv: https://arxiv.org/abs/2407.05040 · HTML: https://mlsanity.com/paper/2407.05040/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 51 references that are in the map.
- First posted on arXiv 6 Jul 2024.

## Builds on (most cited first, 30 of 39)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [WizardCoder: Empowering Code Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2306.08568/) (2023)
- [Instruction Tuning with GPT-4](https://mlsanity.com/paper/2304.03277/) (2023)
- [Estimating Training Data Influence by Tracing Gradient Descent](https://mlsanity.com/paper/2002.08484/) (2020)
- [LESS: Selecting Influential Data for Targeted Instruction Tuning](https://mlsanity.com/paper/2402.04333/) (2024)
- [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022)
- [StarCoder 2 and The Stack v2: The Next Generation](https://mlsanity.com/paper/2402.19173/) (2024)
- [What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning](https://mlsanity.com/paper/2312.15685/) (2023)
- [From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning](https://mlsanity.com/paper/2308.12032/) (2023)
- [Selective Annotation Makes Language Models Better Few-Shot Learners](https://mlsanity.com/paper/2209.01975/) (2022)
- [When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method](https://mlsanity.com/paper/2402.17193/) (2024)
- [Active Prompting with Chain-of-Thought for Large Language Models](https://mlsanity.com/paper/2302.12246/) (2023)
- [On the Effectiveness of Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2211.15583/) (2022)
- [Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuning](https://mlsanity.com/paper/2402.00530/) (2024)
- [RTLFixer: Automatically Fixing RTL Syntax Errors with Large Language Models](https://mlsanity.com/paper/2311.16543/) (2023)
- [Trivial or impossible -- dichotomous data difficulty masks model differences (on ImageNet and beyond)](https://mlsanity.com/paper/2110.05922/) (2021)
- [Rethinking the Instruction Quality: LIFT is What You Need](https://mlsanity.com/paper/2312.11508/) (2023)

## Built on by (most cited first, 20 of 20)

- [A Survey on Data Synthesis and Augmentation for Large Language Models](https://mlsanity.com/paper/2410.12896/) (2024)
- [Type-Constrained Code Generation with Language Models](https://mlsanity.com/paper/2504.09246/) (2025)
- [A Coreset Selection of Coreset Selection Literature: Introduction and Recent Advances](https://mlsanity.com/paper/2505.17799/) (2025)
- [Large Language Models for Code Generation: A Comprehensive Survey of Challenges, Techniques, Evaluation, and Applications](https://mlsanity.com/paper/2503.01245/) (2025)
- [Every FLOP Counts: Scaling a 300B Mixture-of-Experts LING LLM without Premium GPUs](https://mlsanity.com/paper/2503.05139/) (2025)
- [Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspective](https://mlsanity.com/paper/2410.01720/) (2024)
- [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025)
- [Beyond Similarity: A Gradient-based Graph Method for Instruction Tuning Data Selection](https://mlsanity.com/paper/2502.11062/) (2025)
- [SPACE: Noise Contrastive Estimation Stabilizes Self-Play Fine-Tuning for Large Language Models](https://mlsanity.com/paper/2512.07175/) (2025)
- [Uncertainty Profiles for LLMs: Uncertainty Source Decomposition and Adaptive Model-Metric Selection](https://mlsanity.com/paper/2505.07309/) (2025)
- [MapCoder-Lite: Distilling Multi-Agent Coding into a Single Small LLM](https://mlsanity.com/paper/2509.17489/) (2025)
- [Benchmarking Large Language Model Uncertainty for Prompt Optimization](https://mlsanity.com/paper/2409.10044/) (2024)
- [LLM-Aided Customizable Profiling of Code Data Based On Programming Language Concepts](https://mlsanity.com/paper/2503.15571/) (2025)
- [Multimodal Chip Physical Design Engineer Assistant](https://mlsanity.com/paper/2510.15872/) (2025)
- [CODEBLOCK: Learning to Supervise Code at the Right Granularity](https://mlsanity.com/paper/2606.18286/) (2026)
- [ReCatcher: Towards LLMs Regression Testing for Code Generation](https://mlsanity.com/paper/2507.19390/) (2025)
- [Do LLMs Align with My Task? Evaluating Text-to-SQL via Dataset Alignment](https://mlsanity.com/paper/2510.04919/) (2025)
- [Reasoning Distillation and Structural Alignment for Improved Code Generation](https://mlsanity.com/paper/2510.17598/) (2025)
- [Reward-Free Code Alignment from Pretrained or Fine-Tuned LLM: Unpacking the Trade-offs for Code Generation](https://mlsanity.com/paper/2606.28998/) (2026)
- [Fisher-Guided Submodular Data Selection for Continual Pre-Training of Large Language Models](https://mlsanity.com/paper/2610.02593/) (2026)

## Abstract (arXiv)

Recent work targeting large language models (LLMs) for code generation demonstrated that increasing the amount of training data through synthetic code generation often leads to exceptional performance. In this paper we explore data pruning methods aimed at enhancing the efficiency of model training specifically for code LLMs. We present techniques that integrate various clustering and pruning metrics to selectively reduce training data without compromising the accuracy and functionality of the generated code. We observe significant redundancies in synthetic training data generation, where our experiments demonstrate that benchmark performance can be largely preserved by training on only 10% of the data. Moreover, we observe consistent improvements in benchmark results through moderate pruning of the training data. Our experiments show that these pruning strategies not only reduce the computational resources needed but also enhance the overall quality code generation.