# CodeFort: Robust Training for Code Generation Models

arXiv: https://arxiv.org/abs/2405.01567 · HTML: https://mlsanity.com/paper/2405.01567/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 38 references that are in the map.
- First posted on arXiv 11 Apr 2024.

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [StarCoder: may the source be with you!](https://mlsanity.com/paper/2305.06161/) (2023)
- [WizardCoder: Empowering Code Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2306.08568/) (2023)
- [InCoder: A Generative Model for Code Infilling and Synthesis](https://mlsanity.com/paper/2204.05999/) (2022)
- [The Stack: 3 TB of permissively licensed source code](https://mlsanity.com/paper/2211.15533/) (2022)
- [Adversarial Logit Pairing](https://mlsanity.com/paper/1803.06373/) (2018)
- [CodeGen2: Lessons for Training LLMs on Programming and Natural Languages](https://mlsanity.com/paper/2305.02309/) (2023)
- [ReCode: Robustness Evaluation of Code Generation Models](https://mlsanity.com/paper/2212.10264/) (2022)
- [Contrastive Code Representation Learning](https://mlsanity.com/paper/2007.04973/) (2020)
- [Natural Attack for Pre-trained Models of Code](https://mlsanity.com/paper/2201.08698/) (2022)
- [Adversarial Examples for Models of Code](https://mlsanity.com/paper/1910.07517/) (2019)
- [CodeAttack: Code-Based Adversarial Attacks for Pre-trained Programming Language Models](https://mlsanity.com/paper/2206.00052/) (2022)
- [Semantic Robustness of Models of Source Code](https://mlsanity.com/paper/2002.03043/) (2020)
- [Multilingual training for Software Engineering](https://mlsanity.com/paper/2112.02043/) (2021)
- [Adversarial Robustness for Code](https://mlsanity.com/paper/2002.04694/) (2020)
- [Augment your batch: better training with larger batches](https://mlsanity.com/paper/1901.09335/) (2019)
- [Generating Adversarial Computer Programs using Optimized Obfuscations](https://mlsanity.com/paper/2103.11882/) (2021)
- [Adversarial Robustness of Deep Code Comment Generation](https://mlsanity.com/paper/2108.00213/) (2021)
- [ContraCLM: Contrastive Learning For Causal Language Model](https://mlsanity.com/paper/2210.01185/) (2022)
- [One Adapter for All Programming Languages? Adapter Tuning for Code Search and Summarization](https://mlsanity.com/paper/2303.15822/) (2023)
- [CONCORD: Clone-aware Contrastive Learning for Source Code](https://mlsanity.com/paper/2306.03234/) (2023)
- [On Adversarial Robustness of Synthetic Code Generation](https://mlsanity.com/paper/2106.11629/) (2021)

## Built on by (most cited first, 4 of 4)

- [Understanding and Bridging the Planner-Coder Gap: A Systematic Study on the Robustness of Multi-Agent Systems for Code Generation](https://mlsanity.com/paper/2510.10460/) (2025)
- [CCCI: Code Completion with Contextual Information for Complex Data Transfer Tasks Using Large Language Models](https://mlsanity.com/paper/2503.23231/) (2025)
- [Prompt Stability in Code LLMs: Measuring Sensitivity across Emotion- and Personality-Driven Variations](https://mlsanity.com/paper/2509.13680/) (2025)
- [An Initial Exploration of Contrastive Prompt Tuning to Generate Energy-Efficient Code](https://mlsanity.com/paper/2604.02352/) (2026)

## Abstract (arXiv)

Code generation models are not robust to small perturbations, which often lead to incorrect generations and significantly degrade the performance of these models. Although improving the robustness of code generation models is crucial to enhancing user experience in real-world applications, existing research efforts do not address this issue. To fill this gap, we propose CodeFort, a framework to improve the robustness of code generation models, generalizing a large variety of code perturbations to enrich the training data and enabling various robust training strategies, mixing data augmentation, batch augmentation, adversarial logits pairing, and contrastive learning, all carefully designed to support high-throughput training. Extensive evaluations show that we increase the average robust pass rates of baseline CodeGen models from 14.79 to 21.74. We notably decrease the robustness drop rate from 95.02% to 54.95% against code-syntax perturbations.