# CodeACT: Code Adaptive Compute-efficient Tuning Framework for Code LLMs

arXiv: https://arxiv.org/abs/2408.02193 · HTML: https://mlsanity.com/paper/2408.02193/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 41 references that are in the map.
- First posted on arXiv 5 Aug 2024.

## Builds on (most cited first, 30 of 30)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [WizardCoder: Empowering Code Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2306.08568/) (2023)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [StarCoder 2 and The Stack v2: The Next Generation](https://mlsanity.com/paper/2402.19173/) (2024)
- [What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning](https://mlsanity.com/paper/2312.15685/) (2023)
- [From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning](https://mlsanity.com/paper/2308.12032/) (2023)
- [OctoPack: Instruction Tuning Code Large Language Models](https://mlsanity.com/paper/2308.07124/) (2023)
- [CodeGemma: Open Code Models Based on Gemma](https://mlsanity.com/paper/2406.11409/) (2024)
- [#InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Models](https://mlsanity.com/paper/2308.07074/) (2023)
- [Submodular Combinatorial Information Measures with Applications in Machine Learning](https://mlsanity.com/paper/2006.15412/) (2020)
- [Data Diversity Matters for Robust Instruction Tuning](https://mlsanity.com/paper/2311.14736/) (2023)
- [Clustering and Ranking: Diversity-preserved Instruction Selection through Expert-aligned Quality Estimation](https://mlsanity.com/paper/2402.18191/) (2024)
- [Dataset Quantization](https://mlsanity.com/paper/2308.10524/) (2023)
- [Rethinking the Instruction Quality: LIFT is What You Need](https://mlsanity.com/paper/2312.11508/) (2023)
- [Reflection-Tuning: Data Recycling Improves LLM Instruction-Tuning](https://mlsanity.com/paper/2310.11716/) (2023)
- [Exploring the Mystery of Influential Data for Mathematical Reasoning](https://mlsanity.com/paper/2404.01067/) (2024)
- [BugSwarm: Mining and Continuously Growing a Dataset of Reproducible Failures and Fixes](https://mlsanity.com/paper/1903.06725/) (2019)

## Built on by (most cited first, 15 of 15)

- [SWE-Exp: Experience-Driven Software Issue Resolution](https://mlsanity.com/paper/2507.23361/) (2025)
- [SWE-Debate: Competitive Multi-Agent Debate for Software Issue Resolution](https://mlsanity.com/paper/2507.23348/) (2025)
- [ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Use](https://mlsanity.com/paper/2501.02506/) (2025)
- [Open Deep Search: Democratizing Search with Open-source Reasoning Agents](https://mlsanity.com/paper/2503.20201/) (2025)
- [MemGovern: Enhancing Code Agents through Learning from Governed Human Experiences](https://mlsanity.com/paper/2601.06789/) (2026)
- [Hephaestus: Improving Fundamental Agent Capabilities of Large Language Models through Continual Pre-Training](https://mlsanity.com/paper/2502.06589/) (2025)
- [Chain-of-Trigger: An Agentic Backdoor that Paradoxically Enhances Agentic Robustness](https://mlsanity.com/paper/2510.08238/) (2025)
- [TabAgent: A Framework for Replacing Agentic Generative Components with Tabular-Textual Classifiers](https://mlsanity.com/paper/2602.16429/) (2026)
- [Program Analysis Guided LLM Agent for Proof-of-Concept Generation](https://mlsanity.com/paper/2604.07624/) (2026)
- [MOSAIC: Modular Orchestration for Structured Agentic Intelligence and Composition](https://mlsanity.com/paper/2606.00708/) (2026)
- [From Knowledge to Action: Outcomes of the 2025 Large Language Model (LLM) Hackathon for Applications in Materials Science and Chemistry](https://mlsanity.com/paper/2605.03205/) (2026)
- [Programmatic Representation Learning with Language Models](https://mlsanity.com/paper/2510.14825/) (2025)
- [SWE-Spot: Building Small Repo-Experts with Repository-Centric Learning](https://mlsanity.com/paper/2601.21649/) (2026)
- [The Imitation Game: When LLMs Learn to Reason Like Programs via Code-Centric Reasoning Data Synthesis](https://mlsanity.com/paper/2609.16076/) (2026)
- [Exemplar2VQA: A Scalable Exemplar-Driven Visual Question Answering Generation Framework via Multi-Agent Coding](https://mlsanity.com/paper/2609.37655/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have shown great potential in code-related tasks, yet open-source models lag behind their closed-source counterparts. To bridge this performance gap, existing methods generate vast amounts of synthetic data for fine-tuning, leading to inefficiencies in training. Motivated by the need for more effective and efficient training, we propose the Code Adaptive Compute-efficient Tuning (CodeACT) framework. CodeACT introduces the Complexity and Diversity Aware Sampling (CDAS) method to select high-quality training data based on complexity and diversity, and the Dynamic Pack padding strategy to reduce computational resource usage by minimizing padding tokens during training. Experimental results demonstrate that CodeACT-DeepSeek-Coder-6.7B, fine-tuned on only 40% of the EVOL-Instruct data, achieves an 8.6% performance increase on HumanEval, reduces training time by 78%, and decreases peak GPU memory usage by 27%. These findings underscore CodeACT's ability to enhance the performance and efficiency of open-source models. By optimizing both the data selection and training processes, CodeACT offers a comprehensive approach to improving the capabilities of open-source LLMs while significantly reducing computational requirements, addressing the dual challenges of data quality and training efficiency, and paving the way for more resource-efficient and performant models.